Communications training system

ABSTRACT

A communications training system is provided having a user interface, a computer-based simulator and a performance measurement database. The user interface is configured to receive a speech communication input from the user based on a training content and the computer-based simulator is configured to transform the speech communication to a text data whereby the text data can be aligned to performance measurement database values to determine a performance measure of the speech communication. The format of the text data and the performance measurement database values enable the speech communication to be aligned with predefined performance measurement database values representing expected speech communications for that training content.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a Continuation application of U.S. application Ser.No. 15/437,399, filed on Feb. 20, 2017 entitled “SYSTEMS AND METHODS OFPOWER MANAGEMENT”; U.S. application Ser. No. 15/437,399 claims thebenefit of U.S. App. No. 62/296,631, filed on Feb. 18, 2016, entitled“SYSTEMS AND METHODS FOR COMMUNICATIONS TRAINING,” the entire contentsof both are incorporated herein by reference.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

This invention was made with Government support under Contract#W911NF-14-C-0110 and awarded by the U.S. Army. The U.S. Government hascertain rights in the invention.

REFERENCE TO SEQUENCE LISTING, A TABLE, OR A COMPUTER PROGRAM LISTINGCOMPACT DISC APPENDIX

Not Applicable

BACKGROUND OF THE INVENTION

Communication is necessary in any organization to meet the goals of theorganization. For example, in an environment that requires precise andfrequent communication, the U.S. Army faces a number of challenges as itmoves to pursue formation of the Full Spectrum Combat Aviation Brigade(CAB). As the Army moves to this organizational structure, an importantelement is integration of unmanned aircraft systems (UAS) that provideincreased battlefield range and endurance capabilities for both the FullSpectrum CAB and ground units the CAB supports. However, there arechallenges as the Army grapples with issues of integration of UAS intothe CAB, as the role of UAS is rapidly evolving from a traditionalintelligence and surveillance role to a more active participant as ascout-reconnaissance asset that can designate and destroy targets.Accordingly, Manned-Unmanned Teaming (MUM-T) is emerging as a criticalelement of aviation operations. The challenge, however, is that UASoperators traditionally learn few of the scout-reconnaissance skillsappropriate to MUM-T at the schoolhouse. Some of the critical skills notnecessarily learned well at the schoolhouse includes communication andteamwork skills. There is a benefit therefore to provide opportunitiesfor UAS operators to learn these critical skills. Consistent with theArmy Learning Model (ALM), these opportunities may enable learningacross institutional, operational, and self-development domains,necessitating an increasing reliance on novel training tools thatfacilitate practice with respect to communications, with associatedfeedback tools to guide learning. More generally, in the Army of thefuture, such teaming between manned and unmanned assets both on theground and the air will become ubiquitous, necessitating trainingstrategies that build the required skills for mission effectiveness.

BRIEF SUMMARY OF THE INVENTION

The following summary is included only to introduce some conceptsdiscussed in the Detailed Description below. This summary is notcomprehensive and is not intended to delineate the scope of protectablesubject matter, which is set forth by the claims presented at the end.

Embodiments of the disclosed communications training systems and methodsof their use enable an individual student to practice communications andcoordination in individual or teaming scenarios by incorporatingsynthetic entities and natural language processing intosimulators/trainers that emulate voice and chat interactions, withoutrequiring live pilots and other participants. In addition, thesimulator/trainer may be integrated with the performance measuringsoftware. The systems and methods incorporate a system architecture thatpredefines system elements such that training events may be presented toa system user and the system can capture responses in a manner thatallows for accurate understanding of the state of the student andaccurate measurement of the student's performance. The systemarchitecture may include data storage and retrieval format to enablethis measurement. The system may also include processor orcomputer-based simulators that include features for natural languageprocessing that can evaluate student teamwork and communications forcompleteness, accuracy, order, brevity, and timeliness of theinteractions to enhance their learning. The systems and methods may alsoautomate objective performance measurements without the need for liveobservation and may also provide instructors with detailed assessmentsof the student for feedback and after action reviews.

In one example embodiment, the systems and methods may be used to trainan unmanned aircraft systems (UAS) operator to better communicate intheir operational environment.

In some embodiments, the systems and methods may be used to trainworkers that work in a distributed work environment and communicate withother workers over a communications network.

In some embodiment, the communications training systems and methods ofuse allow student interaction and learning through the use of voice anddata messaging within the training environment. The systems and methodsapply the technology of natural language processing modules to enablesimulated or synthetic entities to understand and generate naturallanguage relevant to the training environment. In some embodiments, thesystems may incorporate technical products such as Voisus (for speechrecognition) and Construct (for speech generation) into the trainingenvironment to enable accurate speech recognition, to include speech totext and text to speech, functionality. Utilizing technology thattransforms voice data to text further allows the systems to enhance theperformance measurement capability of the training environment allowingthe student to receive feedback with or without the need for anobserver. The system may also be configured to store training sessiondata for replay by the student and the instructor. The system has a‘gold standard’ of what each expected utterance should sound like asrecorded by an expert. The feedback modules of the system can show theinstructor the progress of the individual trainee or student and alarger group or unit. The feedback module allows instructors to identifytrends, and adjust training curriculum.

Embodiment of the systems and methods provide a rich learningenvironment that allows students to develop their skills without theneed for instructors, live role players or pucksters to support them.This may be accomplished by:

-   -   Applying intelligent synthetic entities that can understand and        produce speech. Enabling students to use the communication        modality that is actually used in practice, spoken language        synthetic entities dramatically increase the fidelity of the        training. In addition, intelligent synthetic entities can        improve the quality of training available to students by        allowing fine-grained control of the training content and        avoiding the inconsistency of human white force role players or        instructors with differing skill levels, different ways of        communicating, and different levels of motivation.    -   Including embodiments of automated performance assessment tools        that can assess both the behaviors and communications of the        student in real time. This assessment data will provide        diagnostic feedback to the student so that they are not only        informed when they performed well or poorly, but also of what        specifically they did that led to the assessment. This        diagnostic performance assessment will allow the student to        effectively learn independently.    -   Using the performance data to select or adapt training in a way        that helps to keep the student engaged and the training content        within the student's zone of proximal development.

In one example embodiment, a computer-based communications trainingsystem is provided comprising a memory configured to store a trainingcontent data set comprising a training event data, the training eventdata defining a simulation data and an event type, a user interfaceconfigured to present the simulation data to a student and receive aresponse data of the student to the simulation data, a communicationplatform configured to receive the response data of the student andtransform the response data to a text data, an interaction managermodule configured to receive the text data to determine an event dataand a measurement environment configured to determine an event measurefor the student based on the event data and the event type. In someembodiments, the response data of the student is a verbal response ofthe student. In some embodiments, the interaction manager module isfurther configured to present an audio data to the student based on thesimulation data and the response data. In some embodiments, the eventmeasure of the student comprises one event measure selected from thegroup consisting of: an accuracy event measure, a completeness eventmeasure, a timeliness event measure, a brevity event measure and anorder event measure. In some embodiments, the response data of thestudent comprises an utterance of the student, the event type comprisesan utterance type and the event measure of the student comprises anutterance type score of the student.

In some embodiments of the computer-based communications trainingsystem, the utterance type comprises one or more utterance slot, theevent measure of the student comprises an accuracy event measure and theaccuracy event measure is determined by the method of: aligning theutterance of the student with the one or more utterance slot whereby oneor more utterance slot score can be determined, and determining theaccuracy event measure for the event type from the one or more utteranceslot score.

In some embodiments of the computer-based communications trainingsystem, the utterance type comprises one or more utterance slot, theevent measure of the student comprises a completeness event measure andthe completeness event measure is determined by the method of: aligningthe utterance of the student with the one or more utterance slot,determining whether the utterance slot is filled or not filled by theutterance of the student, and determining the completeness event measureas a percentage of the one or more utterance slot of the event typefilled by the utterance of the student.

In some embodiments of the computer-based communications trainingsystem, a time between the presentation of the simulation data to thestudent and the receipt of the response data defines an utteranceresponse time, the event measure of the student comprises a timelinessevent measure, and the timeliness event measure is determined bycomparing the utterance response time of the student to an expectedutterance response time.

In some embodiments of the computer-based communications trainingsystem, the utterance type comprises one or more utterance slot, theutterance slot defining one or more brevity terms and the event measureof the student comprises a brevity event measure determined by themethod of: aligning the utterance of the student with the one or moreutterance slot and the one or more brevity terms to determine one ormore utterance slot brevity score and determining the brevity eventmeasure from the one or more utterance slot brevity score.

In some embodiments of the computer-based communications trainingsystem, the utterance type comprises one or more utterance slot in anexpected utterance slot order, the utterance of the student defining aresponse data order, and the event measure of the student comprises anorder event measure is determined by the method of: aligning theutterance of the student with the one or more utterance slot andcomparing the response data order to the expected utterance slot orderto determine the order event measure.

In some embodiments of the computer-based communications trainingsystem, the response data comprises an actual utterance of the student,the event measure type comprises an utterance type, the measurementenvironment comprises a predefined utterance template, the interactionmanager module configured to align the event data of the student to theutterance template to define the utterance slot score as the eventmeasure for the student.

In some embodiments of the computer-based communications trainingsystem, the measurement environment further comprises a performancescore algorithm, a predefined performance scoring data comprising aperformance measure type, the performance measure type corresponding toone or more training events, the performance data comprising an eventmeasure of the one or more training events, and the interaction managermodule is configured to execute the performance score algorithm todetermine a student performance score from the performance data as theperformance measure for the student.

In one example embodiment, a method of providing a performanceassessment of a student in a training simulator is provide, the methodcomprising selecting a training event from a training content, thetraining content comprising a training event, the training contentcorresponding to an expected performance data, presenting the trainingevent to a student, receiving a speech communication as a response ofthe student to the training event, transforming the speech communicationto a text data, aligning the text data to the expected performance datato define an event measure, determining a performance assessment fromthe event measure, and providing the performance assessment to a userinterface.

In some embodiments, the methods and systems for communications trainingare inextricably tied to specifically designed computer-based userinterfaces and specifically designed computer-based simulators thattrain and assess a student's verbal communications against a trainingcontent/scenario and predefined performance measures of verbalcommunication.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

In order that the manner in which the above-recited and other advantagesand features of the invention are obtained, a more particulardescription of the invention briefly described above will be rendered byreference to specific embodiments thereof which are illustrated in theappended drawings. Understanding that these drawings depict only typicalembodiments of the invention and are not therefore to be considered tobe limiting of its scope, the invention will be described and explainedwith additional specificity and detail through the use of theaccompanying drawings in which:

FIG. 1 illustrates a process diagram illustrating the general methods ofone embodiment of systems and methods for communications training;

FIG. 2A illustrates a functional overview of one embodiment of systemsand methods for communications training;

FIG. 2B illustrates an example embodiment of a general architecture fora synthetic entity;

FIG. 3A illustrates a notional architecture for a an example embodimentof an communications training environment;

FIG. 3B illustrates a notional architecture for a an example embodimentof a measurement environment;

FIG. 4A illustrates a state diagram of an example embodiment of oneexample training mission;

FIG. 4B illustrates a state transition table of an example embodiment ofone example training mission;

FIG. 5 illustrates an example embodiment of an architecture tocoordinate dialog state across communication training system components;

FIG. 6 illustrates one example embodiment of a computer system suitablefor a communications training system;

FIG. 7A illustrates a network diagram illustrating one exampleembodiment of the functional element of systems and methods forcommunications training;

FIG. 7B illustrates one example of the different data components oftraining content;

FIG. 7C illustrates one example of the different data components ofperformance measures;

FIG. 7D illustrates one example of the different data components ofperformance dataset;

FIG. 8A shows a process diagram of one example embodiment of methods forproviding communications training;

FIGS. 8B-8F show process diagrams of example embodiments of steps withinmethods for providing communications training;

FIG. 9 shows a Table 1 illustrating a notional example of communicationsby a UAS operator;

FIG. 10 shows a Table 2 illustrating a sample HPML describing a MaximumAltitude performance measure;

FIG. 11A illustrates an example of a slot template showing slots forutterances for each event type;

FIG. 11B illustrates an example of abbreviated legal SME phrases orutterances for a SPOT report event type;

FIG. 11C illustrates an example of abbreviated SME descriptions for SPOTreport with weight and constraint columns defined; and

FIG. 11D illustrates and example feedback table which specifies feedbackmessages according to features and classes of phrases or utterances.

DETAILED DESCRIPTION OF THE INVENTION

Systems and methods for communications training will now be described indetail with reference to the accompanying drawings. It will beappreciated that, while the following description focuses on a systemthat provides training and performance management for UAS operators, thesystems and methods disclosed herein have wide applicability. Forexample, the communications training methods described herein may bereadily employed for computer-based training with any role relying onproper communications such as remote help desk personnel, call-centerpersonnel, air traffic controllers, emergency responders, financial andlegal professionals or hospital personnel. Notwithstanding the specificexample embodiments set forth below, all such variations andmodifications that would be envisioned by one of ordinary skill in theart are intended to fall within the scope of this disclosure.

To incorporate communication skills, the communications training systemsand their methods of use may integrate technology components such asautomated speech recognition (ASR), text to speech (TTS) synthesis, andnatural language processing (NLP). Using these technology components ina computer-based training environment may provide a more natural mode ofinteraction, closely replicating a live mission environment, enablingtraining for critical voice communication skills. Additionally, they mayincorporate technology components such as a performance assessmenttechnology (PM Engine) and performance assessment server applicationtechnology (ASA). These products enable a richer performance feedback(After Action Report (AAR)) and also allow for the simulator/trainersystem to adapt to student performance.

In some embodiments, the communications training systems may includereal time adaptation by enabling the synthetic entities to considerstudent's performance when determining their next behavior or response.In some embodiments, the systems may include intra-scenario adaption tohelp select the next scenario based on a student's past performance. Insome embodiments, selection the next scenario may be provided by themethods and systems disclosed in co-pending U.S. patent application Ser.No. 15/395,574 filed on Dec. 30, 2016 entitled “MACHINE LEARNING SYSTEMFOR A TRAINING MODEL OF AN ADAPTIVE TRAINER” which is hereinincorporated by reference in its entirety. In some embodiments,selection the next scenario may be provided by the methods and systemsdisclosed in co-pending U.S. patent application Ser. No. 14/143,410filed on Dec. 30, 2013 entitled “PROBABILISTIC DECISION MAKING SYSTEMAND METHODS OF USE” which is herein incorporated by reference in itsentirety. In some embodiments, measurement of student performance datamay be provided by the methods disclosed in co-pending U.S. patentapplication Ser. No. 15/098,216 filed on Apr. 13, 2016 entitled “SYSTEMSAND METHODS TO MEASURE PERFORMANCE” which is herein incorporated byreference in its entirety.

Some embodiments of the disclosed systems and methods enable anindividual student to practice communications and coordination inteaming scenarios by incorporating synthetic entities and naturallanguage processing technology that emulates voice and chatinteractions, without requiring live pilots and other participants. Inaddition, the simulator/trainer may be integrated with performancemeasuring technology. The systems and methods incorporate technology,tailored for natural language processing, that can evaluate studentcommunications for completeness, accuracy, order, brevity, andtimeliness of the interactions and enhances their learning. The systemsand methods may also automate objective performance measurements withoutthe need for live observation and may also provide instructors withdetailed assessments and constructive context-sensitive feedback of thestudent performance for after action reviews.

In some embodiments, the systems and methods may be used to traindistributed team members that require verbal communication to accomplishtasks done by different members of the team.

In some embodiments, the systems and methods are directed to train astudent that needs to analyze a situation or receive a communication andrespond (or dialogue) in a structured manner where content, accuracy,completeness, and timeliness is important (e.g. a 911 operator, who musti) ascertain critical information and ii) use it for dispatchingresources). Similarly, this could apply to training air trafficcontrollers for work controlling air traffic in an airport. Other usesof the system and methods could extend to monitoring not just thecontent, but other sensors associated with the communication (such asvolume, pitch, intensity) and even other physical sensors of theoperator (such as heart rate, and galvanic skin response) to indicatethe operator's ability to control and manage their stress levels whilemaintaining accurate communications. The systems and methods may alsoapply to training operators where the receiving systems had limitedunderstanding and ability to parse unstructured communications. Thismight occur in human-machine teaming situations, where a human needs tocommunicate verbally (or textually) with a robot (or team of robots) orother artificial entities. Also, for creating any structured reports(such as weather reports). Also, this could be used when communicatingreports to people who have limited English vocabularies.

In one specific example embodiment, the systems and methods may be usedto train a UAS operator to better communicate in their operationalenvironment. To illustrate this example embodiment, and not forlimitation, to address the US Army's aviation training need, a singleplayer student, simulation based simulator/trainer may be provided thatfocuses on training for UAS operators (see FIG. 2A). Thissimulator/trainer enables the training of skills in a setting where thestudent 201, as a UAS Payload Operator (PO), has a computer-based deviceloaded with the appropriate user interface 202 that is in communicationwith the other training environment elements. In this architecture, thehighly interactive role of the Aircraft Operator (AO) is represented bythe training system through simulated or synthetic entities 250,requiring no human operator. In addition, the student 201 can view hisor her progress without the presence of a human instructor through thecommunications from the user interface 220 or as feedback from theperformance assessment components 284 of the measurement environment.Human instructors can access student records and provide guidance whenneeded, as well. Performance assessments 284 may be assessed at theindividual level as well as at different team and group levels. Forexample, performance may be assessed at a unit or team level or at abattalion level. Performance trends may also be assessed for individualstudents as well as teams or groups as the data for all sessions is inthe ASA and visualized in an instructor dashboard. The student interactswith the training environment's client interface 220. In one embodiment,the client user interface 220 is comprised of a user portal for loggingin, maintaining and viewing records, and accessing the simulator itself.The simulator user interface 220 may be specifically programed toclosely resemble the configuration of tactical windows found in theactual equipment to be trained. The computer-based simulator may alsohave a specialized user interface device, as described below, to provideuser input into the simulator/trainer. The computer-based simulator maybe comprised of a client as a user interface 220 and an applicationbackend to provide the other functions of the simulator and simulationenvironment. The client and backend may be hosted together in astandalone configuration or the application backend, such as a serverand/or database, may be hosted over a network. Simulator and trainingresults may be captured in a performance database that the student,instructor or other system component may access.

As used herein, the terms “module,” “platform,” “environment,” and“engine” refer to hardware and/or software implementing entities anddoes not include a human being. The operations performed by the“module,” “platform,” “environment,” and “engine” are operationsperformed by the respective hardware and/or software implementations,e.g. operations that transform data representative of real things fromone state to another state, and these operations do not include mentaloperations performed by a human being.

One Embodiment of a Communications Training System

In one example embodiment, the communication training system generallyprovides a way to make a performance assessment of a student in atraining simulator by: presenting a training event to the student in thetraining simulator, receiving a speech communication as a response ofthe student to the training event, transforming the speech communicationto a text data, comparing the text data to a predefined simulationdataset to define a performance data and comparing the performance datato a predefined performance measure dataset to define the performanceassessment of the speech communication. In some embodiments, the systemfurther comprises elements to transform the text data to an HPML datawherein comparing the text data to a predefined simulation datasetcomprises comparing the HPML data to a predefined simulation dataset. Insome embodiments, the response of the student is used to define anothertraining event to the student. In some embodiments, the presentation,response and assessment steps are iterated using multiple trainingevents with multiple responses are used to make a performance assessmentof the student.

The training environment generally represents the environment and systemcomponents the student interfaces and interacts with when receivingtraining. The training environment also collects data from the student'sperformance. The training environment may comprise a simulationenvironment with interfaces providing input from a student as well asoutput to the student. The simulation environment generally alsocomprises one or more synthetic entities and training content/scenarios.The synthetic entities generally communicate with a communicationplatform, a simulation dataset and an interaction manager module. Fromthe student's performance with the simulation environment, the trainingenvironment tracks and provides a performance dataset for use by theperformance measurement engine.

In some embodiments, a measurement environment is provided to providethe data for analyzing the performance dataset. The measurementenvironment generally comprises defined performance measures or scores,functions to assess the performance of the student against theperformance measures and functions to provide feedback based on theperformance assessment.

For illustration purposes and not for limitation, one example embodimentof a communications training system is shown in the high-levelfunctional diagram of FIG. 1 . As shown, the communications trainingsystem 100 generally comprises a system utilizing a training environment110 and a measurement environment (PM Engine) 180B. The trainingenvironment 110 generally comprises the system components that providethe interaction with the student through a user interface 120 to thesimulation environment 140 and the measurement environment 180B. Thesimulation environment 140 generally comprises the system componentsthat define and execute the processes representing the simulation to thestudent. Within the simulation environment 140, synthetic entities 150are provided that are also in communication with a trainingcontent/scenario database 142. The synthetic entities 150 generallycomprise a communication platform 152 that interacts with a trainingcontent data 142 and an interaction manager module 158. The interactionmanager module 158 generally uses data from the training content data142 and the student's performance data 166 to determine how the studentis performing and what should be presented to the student next. Thesimulation environment 140 may also be configured to communicate with ameasurement environment 180A within the training environment 110.

The measurement environment 180B generally comprises the systemcomponents that provide measurement, assessment and feedback of thestudent's performance. The measurement environment 180B is distributedor separated from the training environment 110 but is in communicationthrough a proxy (PM Engine proxy) 164. The measurement environment 180Bmay comprise components to include performance assessment module 184, aperformance feedback module 186 and a performance measures database 182.In some embodiments, an access application module 162 is provided toprovide access to measurement environment modules and the performancemeasures database 182.

The communications training system 100 may operate as a distributed,web-based simulation environment supporting multiple, simultaneousplayers executing unique missions. These system components are describedin further detail below.

FIG. 3A also shows a system architecture of an example embodiment of acommunications training system specific to a training environment for aNVTT training simulator. The communication platform, here Voisus 352,comprises representations for each synthetic entity 350 type andsupports speech to text and text to speech interactions. As shown,speech to text is indicated by the Automatic Speech Recognition (ASR)module to Natural Language Unit (NLU) module relationship and text tospeech is represented by the Natural Language Generation (NLG) module toText to Speech (TTS) module components (Voice recognition server (ASTiVoisus)). The simulation environment, here NVTT Simulator 340distributed from Voisus 352, provides the simulation data (event andscenario content) including the student's user interface (e.g., audiocommunication) and the air vehicle (e.g., visual stimulus) simulation.The NVTT simulator 340 contains proxies for interfacing to othertraining system architectural components as well as a PM Engine proxy364. The NVTT simulation data 357 functions as the simulation datasetand the mission execution data 359 functions as the execution dataset.The One Semi-Automated Forces (OneSAF) provides additional simulationdata such as contextual elements of the simulation including the terrainand simulated or synthetic entities on the terrain. These entities areobserved and heard by the student playing the simulation. OneSAFincludes the Night Vision Image Generator (NVIG). Computercommunications between components may be made through a digitalcommunications network such as the Internet or through any other type ofcommunication network. For this embodiment, the training environment 310is in communication with the measurement environment 380 through the PMEngine proxy 364, the measurement platform controller 383 and the PMEngine pool manager 385. The measurement platform 380C is a grouping ofsystem components to include the performance dashboard 386 that providesthe performance feedback functions, ASA 362 provides access to theperformance measures database 382 and the PM Engine 384 provides theperformance assessment functions.

Simulation Environment.

Referring to FIG. 1 , the simulation environment 140 generally comprisesthose system components that define and execute the processesrepresenting the simulation to the student. The simulation environmentgenerally comprises a computer-based simulator having synthetic entities150 and training content 142. Within the simulation environment 140, thesynthetic entities 150 generally comprise the hardware and/or softwareimplementing components that represent the entity that the studentinteracts with in the simulation environment 140. The trainingcontent/scenarios 142 are predefined data sets that define the trainingtasks and the simulation data necessary to present the content to thestudent through a user interface 120. The training content 142 alsodefine the tools and data sets that are to be compared to the student'sresponses to determine the student's state and determine how theyprogress through the simulation.

User Interface.

Referring to FIG. 1 , the user interface 120 may be any type ofinterface or device that allows the user to provide input to thesimulation environment 140. In some embodiments, the use interface 120may include a microphone to receive speech communications and a speakeror other device capable of simulating noise and speech communication. Insome embodiments, the simulation interface 120 is a graphic userinterface (GUI) configured to match the environment to be trained. Forexample, in one embodiment, the GUI reflects the actual UAS groundcontrol station. The user interface 120 is intended to achieve aneffective and realistic integration of communications user interfacesinto the overall user experience. The user interface 120 may alsoinclude After Action Report (AAR) capability to include methods forpresenting additional performance measures to the student, and enablingthe review of communications made during the training. The userinterface 120 may also include an instructor dashboard to represent theresults and trends of the performance of multiple students.

In some embodiments, the user interface 120 to the simulated environment140 may be through various specialized hardware communications devices,live communication devices or simulated communication devices presentedthrough software clients and graphic user interfaces. These clients mayprovide networked voice communications on a variety of platforms toincluding PCs, tablets, generic hardware-based platforms or specificallydesigned platforms. These clients may support features such as simulatedradio nets, intercom channels, realistic radio effects, point-to-pointand conference calling, text chat and live radio communications andcontrol.

Synthetic Entities.

As shown in FIG. 1 , the synthetic entity 150 generally comprises thesimulation or the entity that the student interacts with in thesimulation environment 140. For example, the simulated entity may beanother party in the scenario that the student has to communicate with.The other party may be represented by a set of predefined communicationsin system databases that are presented to the student to representactivities of that party in the scenario. To support realistic training,the specific decisions and actions the student makes should have anobservable effect in the simulation environment. To support this, thesynthetic entities 150 can reflect these decisions and actions. Forexample, the synthetic entity 150 may include synthetic operator voicetechnology as a stimulus and to produce speech responses to the student.The synthetic entities 150 may also be able to result in performancedata that more accurately reflects detailed performance measures.

As shown in more detail at FIG. 2B, one embodiment of the syntheticentity 250 generally comprises a processor or computer-based tool ortechnology, such as a computer software module (for example Constructand Voisus), that receives and provides communication with the student(for example voice communications) and an interaction with or includesthe interaction manager 258. As shown, Voisus 254 provides ASRfunctionality and Construct provides TTS functionality 255A and NLPfunctionality (here shown as natural language understand (NLU) 255B).The synthetic entities 250 also may include or may be able tocommunicate over a distributed network such as DIS with a trainingcontent dataset (not shown) to provide training content/scenarios to thestudent and may be configured to communicate with a measurementenvironment such as performance engine 280.

Some embodiments of the synthetic entities may incorporate TTS and ASRfunctionality and technology similar to the transcription of voicecommunications systems and methods disclosed in US Pub. No.2015/0073790, published on Mar. 12, 2015, U.S. application Ser. No.14/480,388, filed Sep. 8, 2014 and herein incorporated by reference inits entirety. In some embodiments, the TTS functionality may include thefeatures of the Construct tools as marketed and offered by AdvancedSimulation Technology Inc. (ASTI) of Herndon Va. under the name of“Construct” and the ASR functionality may include the features of toolsmarketed and sold under the name of the “Voisus” product also offered byASTI. The TTS and ASR functionality may include automated voiceinteractions including functionality to automate calls for interactionssuch as for: Air Traffic Control; Close Air Support/Call for Fire, GMDSSCommunications; Medevac; and NBC reports. Construct automates theinteractions to provide a hands-off tool for instructors, which reducesworkload and role-playing demands. This functionality may reproduce thecontent and behavior of real-world communications inside the trainingsystem. The synthetic entities may follow the training communicationsplan and feature radios with realistic cryptography, propagation, anddistortion effects. Students may be able to tune to multiplecommunications nets, each with mission-customized andcontextually-accurate radio traffic. Students may be able to conversewith the synthetic entities face-to-face in 3D event environments orover simulated radios. The TTS and ASR functionality may preventnear-empty airwaves and manpower-intensive role-playing in thecomputer-based simulator by creating intelligent entities that interactverbally with students. In some embodiments, the TTS and ASRfunctionality allows 3D positioning of voice and radio transmissions andmay provide realistic radio noise and distortion effects. The additionof ASR, TTS, and NLP technologies further produce an environment thatprovides significant independence and flexibility for the student.

Training Content.

Referring back to FIG. 1 , the training content 142 of the simulationenvironment 140 generally comprises the training events and situationspresented to the student through the user interface and thecomputer-based simulator. In general, the training content is apredefined data set stored in a memory of the training simulator andcomprises a simulation dataset 157 and an execution dataset 159.

The training content may be based on Training Missions and CampaignMissions. A Training Mission is a short exercise or training event topractice a small task (i.e. a 2-minute exercise to turn the radio to thecorrect channel, and to contact the ground observers, or maybe give afive line report). Training Missions may be introductory sessions on howto play the simulation or small tasks or events within larger TrainingMissions. A Campaign Mission is a longer (open-ended) scenario with amission brief (i.e. things to accomplish like reconnoiter a large area,or find a downed plane) (or several things to do). In a CampaignMission, the student might need to use several of the skills learned inthe Training Mission. Campaign Missions are typically scored and do havefeedback and have multiple events within them. Both the Training andCampaign Missions may be defined to be structured and reflective ofrealistic scenarios that support progressively more difficult trainingtasks. The methods for communications training may enhance existingscenarios and develop new scenarios to allow the student to train usingASR, TTS, and NLP capabilities.

Examples of training content include training events to train voice FMradio communications with scenario entities; to perform aerialreconnaissance and report on ground assets and activities; de-conflictairspace; conduct target hand-over to ground units; call for and adjustindirect reconnaissance zones; and request and designate targets forfurther reconnaissance.

Additional training content may include scenarios with more diversereconnaissance zones that include multiple zones. Some embodiments mayuse pre-formatted communication menus with drop down fields.

The objective for each training event is to connect the current skill ortask in the simulation environment with a communication skill usingspeech recognition. These training events or scenarios may be mapped toa list of critical communication skills. The proper communication skillsand formats can be predetermined by subject matter experts (SMEs).

As shown in more detail at FIG. 7B, the training content includes atraining event data 742A comprising simulation data 757 and executiondataset 759. The simulation data 757 generally comprises anidentification or definition of the simulation data needed for thesimulator to perform the training event (e.g., text, audio or imagerydata) and the execution data 759 generally comprises the data and toolsnecessary to analyze the response data from the student to determine howthe student should progress through this event and the simulation. Thetraining event data 742A may also include performance measures 742B forthe training event or an identifier to access a distributed source ofthe performance measures for the training event.

Simulation Data.

Referring to FIG. 1 , the simulation data, or simulation dataset 157 isused by the synthetic entities 150 to receive and provide feedback tostudents when presented the training content/scenarios. Simulation data157 generally comprises the actual data, or an identification of theactual data so that it can be retrieved from another data source, neededby the interaction manager 158 to communicate events from the simulationto and from the simulation environment 140 and the user interface 120.For example, the simulation data 157 may comprises a tag or reference tocall simulation data such as the image or communication data requiredfor the training event. The simulation dataset 157 includes data toreflect the typical interactions between students and other assets thatmay be needed from the simulation environment 140. For example, typicalinteractions may include a textual representation of a spoken word orphrase and a corresponding meaning of the phrase. The simulation dataset157 may reflect the true variation in speech and actions that isobserved in actual training. The simulation datasets 157 may cover allthe training scenarios of interest, including common missions as well ascommon lower-level events within missions. The simulation dataset 157may include all relevant vocabulary and phrasings from the domain. Moregenerally, the simulation dataset 157 may include variations in studentutterances relative to the same stimuli (either spoken stimuli or eventsin the simulation). In one embodiment, the dataset 157 covers a varietyof levels of student expertise and levels of performance. Examples ofoptimal performance, in particular, provide reference exemplars fordeveloping and testing communications measures.

To develop and predefine the simulation dataset for the simulationenvironment, a representative dataset of student interactions may beused. A preliminary simulation dataset may also be comprised ofsynthesized dialogs that capture typical interactions between studentsand other assets in the environment. The simulation dataset may alsocomprise both synthetic data (manually developed) and actual data fromtraining events. These simulation datasets may be used as developmentsimulation datasets to help tune the speech recognition toolkit, tailorthe natural language processing (NLP) module to the domain, develop thedecision-making approach of the interaction manager, and implement thecommunications performance measures.

While extremely useful for initial development, these simulationdatasets may still lack the true variation in speech and actions that isobserved in actual training. In one embodiment, a preferred simulationdataset may be constructed from role-playing of students with whiteforces, or from student interactions recorded in other relevantsimulation environments. If such data is difficult to acquire, data mayalso be derived from “Wizard-of-Oz” sessions comprised of a few studentsinteracting with development team members playing the roles of syntheticentities in the simulation.

For development, the simulation dataset can be transcribed. Annotationmay be helpful for development of some components. For example, for theNLP and interaction manager components, categorization of studentutterances into domain-specific dialog acts may be required.

In some embodiments directed to UAS training, the simulation dataset maybe based on an elucidation of the communications doctrine andphraseology used by UAS operators when communicating with Air MissionCommanders. The dataset may be used to tune the speech recognitionlanguage model to recognize UAS operator radio calls with high accuracyand to create speech templates for radio calls.

Execution Data.

Referring to FIG. 1 , the execution data or execution dataset 159generally comprises the data used by the interaction manager 158 toevaluate the response data within the training event and determine theoutput of the simulation environment 140 to that response. The executiondata 159 generally comprises software and mathematical models to analyzethe response data. In one embodiment, the execution data 159 definesstate machines and state transition tables configured to take thecurrent state and the input data and compare that data to a set oftransition data to determine whether the student should transition tothe next state. The state transition tables may contain predeterminedvalues reflecting the inputs, states and updated states (as transitionsor outputs) specific to that task or training scenario. In someembodiments, the state transition table may recognize a range ormultiple types of input data to map to or otherwise align with aparticular updated state and a corresponding transition. The executiondataset 159 may also include the state machines and tables that may beused by the interaction manager 159 to determine the state of thesynthetic entity 150 based on the response data from the student.

FIG. 4A illustrates an example of a state machine reflecting the statetransitions through a training event. FIG. 4B illustrates and example ofa state transition table generally comprising the predefined datapopulated in the state transition table to be the result of thecomparison of the response (input) and state data to determine whetherthe state of the student should change. This transition data isformatted to be easily aligned or otherwise compared to the text data asgenerated from the actual response data.

Communication Platform.

Referring to FIG. 1 , the communication platform 152 generally comprisesthe system components that provide the functionality providing thestimulus to the student as well as the functionality to receive andcommunicate the student's response to other components of the trainingsystem. For embodiments that verbally interact with the student, thecommunication platform 152 supports speech to text and text to speechinteractions. As shown, speech to text is indicated by the AutomaticSpeech Recognition (ASR) module 154 to Natural Language Understanding(NLU) module 156A relationship. Text to speech is represented by theNatural Language Generation (NLG) module 156B to Text to Speech (TTS)155 module relationship.

Automated Speech Recognition (ASR) Module.

Referring to FIG. 1 , the automated speech recognition module 154provides a customized speech recognition language model that transcribesstudent communication with high accuracy. In some embodiments, thetranscription is in real-time or near real-time virtual multi-channelradio panel to communicate with simulation entities to affect aircraftmaneuvers, weapon fires, support calls, and reconnaissance reports.Synthetic entities 150 may respond with real-time feedback based onspeech and context presented by the simulation environment 140. Thesimulation environment 140 allows the student to select which channelsto transmit and receive on, ensuring that h/she remain aware of whichautomated entity they are communicating with. The communication platform152 utilizes automatic speech recognition (ASR) modules 154 tuned withapplication specific data followed by a natural language understanding(NLU) modules 156A that extract relevant meaning from the text. Multiplepotential utterances can be collapsed into a single “extracted” meaning.For example, a student utterance of “Fly east” may have the sameextracted meaning, as “Fly heading 090”. The extracted meaning, based onboth the content of the student transmission and the current simulationcontext within the mission, is used for real-time feedback as well aslater performance scoring. The natural language generation module (NLG)module 156B and a text to speech (TTS) module 156 library generatesrealistic transmissions for real-time vocal responses to the student viaa variety of doctrine and phraseology for utterance types such as SPOTand BDA reports, remote hellfire, and other radio interactions. Themodules may have natural language capability and, hence, may be able toaccurately transcribe even highly variable student speech. This mayincorporate application specific phraseology, geography, call sign, andother data over the period of the scenario. The modules may alsoleverage SMEs to prioritize the phraseology and quickly provide tuningfor the most used and most important communication. Collectedtranscripts and recordings (from the training system itself, othersimulator/trainers, and real world operations, as available) may also beused in tests to determine the performance level of the speechrecognition system and confirm continued accuracy improvements duringthe project.

In one embodiment, the Voisus product line of applications provides theunderlying communication capabilities for this solution. A Voisus moduleor server acts as part of an embedded communications platform in thesimulation environment providing intercom and radio capabilities to allstudents in the training system. Students wear USB headsets as a userinterface that plugs into a computer-based device such as a Windows orLinux PCs. A variety of user interfaces are available with this productline, from physically realistic hardware radio panel devices, to smallfootprint software interfaces for Windows and Linux PCs, and web-browserbased interfaces. In this embodiment, the Construct application runs ona Voisus server platform and provides ASR and TTS capabilities. The ASRand TTS capabilities embedded in Construct are adaptable to a variety oftraining applications, including close air support, call for fire, andair traffic control. With Construct, each synthetic entity is able tolisten and transmit on its own simulated radio, with phraseology andbehaviors that closely match real world operations. A modular, pluginbased architecture supports adaptation for new applications. Usingphraseology data collected in the simulation dataset, the Voisusproducts may be configured to create a customized speech recognitionlanguage model that transcribes UAS radio calls. The language model mayhave natural language understanding capability and, hence, is able toaccurately transcribe even highly variable student speech. The systemmay prioritize the phraseology to recognize the most used and mostimportant communications. Collected radio call transcripts andrecordings (from the training system itself, other simulator/trainers,and real world operations, as available) may be used to determine theperformance level of the speech recognition system and confirm continuedaccuracy improvements. The speech recognition systems may also be usedto extract information and meaning from the speech transcripts.

Natural Language Understanding (NLU) and Generation (NLG) Modules.

Referring to FIG. 1 , the NLU module 156A converts the automated speechrecognition output into representations that are communicated and usedby (1) the interaction manager module 158, which decides on the nextaction to take in the scenario, and (2) the measurement environment180A, which assesses the correctness of student utterances. Thecommunicative requirements of the synthetic entities 150 may span therange of highly structured communication to relatively unstructuredcommunication. The NLG module 156B utilizes information such as actionand state information from the interaction manager to select andprovides the particular dialog and communication to be presented to thestudent with the TTS module 155.

One challenge of understanding the communications made within thesimulation environment is the high likelihood of speech recognitionerrors. When the training of the student is low, the variation betweenstudent's speech is likely to be greater, posing a challenge for thespeech recognition system. Moreover, the more unstructured thecommunications, the greater the chance that the speech recognizer'slanguage model will falter, increasing the speech recognition errorrate. To handle the range of communications in typical scenarios andinevitable speech recognition errors, the NLU module 156A does not relyon spotting keywords or extracting specific phrases. Instead, NLU module156A may take a layered approach that leverages linguistic patterns,supervised machine learning, and deeper linguistic analysis asnecessary.

In one example embodiment, a pattern-matching platform may be used fortext analytics with the NLU module. The pattern-matching platform mayintegrate natural language processing components for collecting,processing, and analyzing text data for a variety of domains, includingperformance assessment, social analytics, and intelligence analysis. Insome embodiments, the pattern-matching platform may provide tools forboth intelligent dialog agents and communications analysis. Thepattern-matching platform may have dialog agent modules that integraterule-based and supervised machine learning approaches to naturallanguage understanding and dialog management. The pattern-matchingplatform may have communications analysis modules to assess theperformance of individual students, teamwork, and multi-team systems.The pattern-matching platform may provide content analysis—whatoperators discuss and how they discuss it—as well as structural analysiscommunication networks and their dynamics.

For simple student utterances, pattern-based information extractiontechniques are sufficient. The NLU module decides whether a studentutterance can be handled by domain-specific pattern templates with highconfidence or requires further analysis. The NLU module performs asurface analysis of the utterance, filtering by features that mightindicate a simple utterance, and performs dialog act recognition.

For more complex utterances and to account for speech recognitionerrors, the system may employ machine-learning based text classificationmethods to augment the understanding pipeline. As needed, for morecomplex or unstructured utterances, the NLU module can invoke deepersyntactic and semantic analysis.

Specific to one embodiment, Table 1 shown in FIG. 9 depicts someexamples of the complexity of increasingly unstructured communicationsby the UAS operator role that the system should adequately comprehend,focusing on air-to-air communications between the UAS operator and amanned helicopter pilot. For example, acknowledgements of utterances(e.g., commands) are generally straightforward to understand as theyrepeat the syntactic and semantic structure of the manned aircraftsynthetic entity's utterance. This makes it easier to classify theutterances and extract the necessary information for understanding withhigh precision.

Accurate comprehension of another utterance type, such as SPOT reports,is somewhat more challenging, however. First, the system must segmentand classify the utterances accurately. For example, in the example ofitem 3 in Table 1 of FIG. 9 , the description of the building undersurveillance spans two sentences, requiring cross-sentence informationintegration. Second, there is some leeway for linguistic variation inhow each aspect of a SPOT report is reported, requiring an understandingsystem with greater coverage of the linguistic devices for encodingsemantic content for the domain. In still other situations,communication between the UAS and manned aircraft can become relativelyunstructured, making understanding (and responding) significantly morechallenging. For example, in item 4 of the table, the UAS asks aquestion of the manned aircraft pilot. Enabling the NLG system torespond to questions requires accurately grasping the focus of thequestion—often requiring grounding of phrases with objects in theenvironment, as here—and crafting a sufficient response.

Text to Speech (TTS) Module.

Referring to FIG. 1 , the TTS module 155 provides a speech response tothe student based on the input provided from the user interface 120 andthe results from the interaction manager 158 given the analysis of theresponse from the ASR module 154 and the NLU module 156A. The NLG module156B typically provides the input to be used by the TTS module 155 tocommunicate an event to the student.

To predefine and develop appropriate communications responses fromsynthetic entities, the responses may be defined from a communicationplan. For example, for UAS training environments the communication planmay include relevant radio nets such as team internal nets (frequencies)and so on expected for that environment. The TTS responses may becustomized with pronunciations for the specific geography and waypointnames, call signs, and other terminology to ensure quality speech isheard by the student. The TTS module may customize communication soundeffects including crypto and background sounds mixed into communicationtransmissions, as appropriate, to create realistic syntheticcommunication transmissions. Speech templates may be predefined forradio requests, responses, and acknowledgements from external assets inorder to simplify natural language generation. These templates may beparameterized for variables like headings, altitudes, and waypoint namesto be insert into the generated speech on the fly.

Interaction Manager Module.

Referring to FIG. 1 , the interaction manager module 158, also calledthe interaction manager, is a module that generally provides theintelligence of the synthetic entity 150 and the communication trainingsystem 100. The interaction manager module 158 takes the input from thecommunication platform 152 (text data representing the response of thestudent), determines the progression of the state of the student and thesimulation based on the response provided by the student as compared topredefined state transitions and determines what action should next bepresented to the student.

To maintain the state of the student's interaction with the scenario,the interaction manager module 158 represents the student interactioncontext, including both spoken interaction and actions in thesimulation. In some embodiments, maintaining state about spokeninteraction requires a history of the utterances so far, from bothstudent and synthetic entities. And a corresponding action historyrepresents the progress of the student in carrying out the mission. Theinteraction manager may also have knowledge of the mission and theindividual tasks required to complete the mission.

The interaction manager module 158 decides on a next action in thesimulation, given the state of the interaction and the pedagogical goalsof the scenario. The action can be a spoken interaction and/or asimulated physical action carried out via one of the scenario'ssynthetic entities. If the action is a verbal response, the interactionmanager selects and provides a template for the response the NLG module.The NLG module combines the interaction manager module's selected actionand information from the current state representation to craft aresponse to the human user. The result of this action is a newinteraction state.

To perform its functions, the main submodules of the interaction managermodule 158 span the maintenance of state and action selection: (1)interaction context and (2) framework for action selection. Theinteraction context integrates information from the “world state” of thesimulation with information about the spoken interaction of the studentand the synthetic entities. The interaction context submodule will trackthe entities and events mentioned in the training scenario so far, andmaintain a history of the spoken interaction, including the syntacticand semantic structure of each student utterance. The interactioncontext submodule is also responsible for directly tracking discourseobligations, such as the need to respond to a question from the student.The framework for action selection operates over the interaction staterepresentation. To ensure that the interaction manager's actionselection meets the requirements for interactivity that produce thedesired training outcomes, several possibilities for decision-makingframeworks may be used. One approach is a finite-state automaton-basedalgorithm, in which interaction states are mapped to actions in thescenario. While easy to implement, finite-state approaches have limitedcapacity to express the state of the interaction and possible actionsbased on these states. For moderately complex interaction scenarios, aricher representation of state is typically required. Action selectionover such state representations can take the form of update rules overthe state, based on a general or specific policy for action-taking. Insome embodiments, the decision-making algorithm is a state-machine withstate machine tables. Errors in speech recognition and languageunderstanding may introduce significant uncertainty into the student'sinputs. Hence, to account for uncertainty methods for interactionmanagement based on statistical machine learning may also be used.

Performance Data.

Referring to FIG. 1 , the performance dataset 166 generally comprisesthe response data of the student, such as text utterance data of thestudent, formatted in the expected forms for each training event. Theperformance data 166 may also include other meta-data associated withthe response such as the time of the presentation of the training eventto the student and the time the student provided their response. In oneembodiment, the text data these are bundled into JSON objects. As shownin FIG. 7D, the performance dataset may also comprise score data of thestudent to the training events.

Measurement Environment.

Referring to FIG. 1 , the measurement environment 108B provides anassessment of student performance using data from the other systemcomponents as well as measures developed specifically for thecommunications training system. The measurement environment 180Bgenerally provides this assessment using a performance assessment module(PM Engine) 184. The measurement environment 180B may also compriseother components: a performance feedback module (Performance Dashboard)186 and an access application module 162 to store and access measurementenvironment information (A-Measure Server Application or ASA) such asinformation in a performance measure database 182. Integration of themeasurement environment 180B components enhance the performanceassessment module 184 and performance feedback module 186 capabilitiesof the communications training system training environment 110 byintegrating the relevant human performance measures for use by bothsynthetic entities 150 and performance feedback modules 186. Themeasurement environment 180B may be distributed from the trainingenvironment 110 and in communication via a proxy such as the PM Engineproxy 164 as shown. In some embodiments these features may residelocally in the training environment shown here as measurementenvironment 180A.

One embodiment of a measurement environment is shown in more detail inFIG. 3B. This measurement environment 380B comprises the measurementplatform controller, the PM Engine pool manager 385, the performancedashboard module 386, the PM Engine 384, and the ASA 362. In thisembodiment, the ASA 362 includes the performance measures database. Theperformance dashboard module 386 generates scorecard and AAR displaysfor viewing visualizations of the performance data collected andprocessed by PM Engine 384 and stored within the ASA 362. These displaysmay be dynamically generated using the configuration informationcontained within the ASA 362. This includes information about thefilters, data sources, and visualization the dashboard creator wantsdisplayed in the user interface.

The Performance Assessment (PM Engine and PME) and Access Application(A-Measure Server Application (ASA)) Modules.

Referring to measurement environment 180B of FIG. 1 , the performanceassessment module 184 serves as the data analysis component of themeasurement environment 180B. Also called the PM Engine, thisperformance assessment module 184 is an analysis tool that can query,filter and perform calculations on any type of data. The performanceassessment module 184 calculates event and performance measures andassesses measures from the event data generated during the execution ofthe simulation and serves as the data analysis component of themeasurement environment 180B. The performance assessment module 184performs the comparison, or alignment of event data from the simulationenvironment 140 with performance measurement data 182 to score thetraining event response data as compared to expected event responsedata. In some embodiments, the performance assessment module 184 usesHuman Performance Markup Language (HPML) to handle data from a varietyof sources, such as a distributed simulation network (e.g., HLA or DIS),log files, and Aptima's SPOTLITE™. More specifically, the performanceassessment module 184 processes instructions defined in HPML; connectsto simulations, physiological devices, mobile devices and/or other datasources; subscribes to the necessary data streams in the trainingenvironment; uses that data to measure and assess performance; andoutputs the resulting measurement and assessment data in real-time. Theperformance assessment module may be configured with a plug-in interfacethat allows it to integrate with many different data sources andmeasurement result repositories for increased flexibility in thecalculation and distribution of performance data.

For the communications training system, the performance assessmentmodule processes event and performance data consistent with SPOT, CCA,BDA, IF and THO Report events and calculates corresponding event scoresand performance measures. The performance assessment module may alsoassess event data on different dimensions including accuracy,completeness, order, brevity and timeliness and duration. Scores andmeasures may be assessed on any type of rating scale and in someembodiments the rating scale is a three tier discrete stoplight ratingscale (e.g., red, green and yellow).

The access application module 162, also called the A-Measure ServerApplication (ASA), is an application for storage, retrieval, management,and analysis of performance measurement data from the performancemeasures database 182. It is comprised of an application providingaccess to a relational database storing performance measurement data anda set of RESTful services which allow for the interaction with thatRESTful data. The ASA may also use the HPML format to define performancemeasures, as well as to describe human performance from a variety ofsources, including PM Engine™ and SPOTLITE™

The performance measures database may be hosted using an SQL Server anduse Entity Framework as the ORM tool providing the data model. Webservices which require user authentication are used to store andretrieve data with the ASA. Data stored through the ASA is used to driveboth real-time and AAR web based performance dashboards.

Interfaces support communication between existing training environmentcomponents and both the PM Engine and ASA to provide performanceassessment capabilities of the training system. This is provided by anintegration layer between the PM Engine and the training environment.This integration layer serves two primary purposes: it allows thesimulator and communications data to be consumed by the PM Engine; andit allows the performance measurement results produced by the PM Engineto be made available to the training environment for consumption bysynthetic entities or other modules.

The integration layer may be composed of a PM Engine proxy for thetraining environment and a connecter plug-in for the PM Engine. The PMEngine proxy may be a module within the training environment thatprovides communication with the performance dataset, including thesimulator and communications data necessary for performance assessment,to the PM Engine and receives the performance assessment/measurementdata produced by the PM Engine. The connector plug-in for the PM Enginewill both consume the simulator and communications data provided by thePM Engine proxy, and publish measure results back to the PM Engineproxy.

The ASA may also be integrated into the training environment through alocal measurement environment. The ASA is used to collect and storeperformance data during simulation runtime and subsequently serve as theperformance data to the performance feedback module. The ASA may enablethe performance feedback module to work off of performance data storedin the ASA via the RESTful service APIs that it exposes.

Performance Measures.

Performance measures are data used to collectively assess both thebehaviors and communications of the student. The performance measure maybe provided in real time and may be implemented in HPML. The performancemeasures can be used both to populate the AAR and to provide thefoundation for performance based adaptation of the simulationenvironment.

Human Performance Markup Language (HPML) is an XML-Schema-based languageintended to cover all meaningful aspects of human performancemeasurement in various training and operational environments. The HPMLhierarchy enables the representation of both generic concepts (e.g.,measurements and assessments) and mission specific concepts (e.g.,instances of measurements and instances of assessments) necessary forcapturing the experiences associated with human performance and humanbehavior. Specifically, it is an XML based language designed to expressperformance measurement concepts in a format that is both machine andhuman readable. It enables the explicit combination and transformationof performance data into performance measurements and assessments. Thisallows measures to be constructed independent of any specific trainingor operational system. At a basic level, the performance measurementinstructions defined in HPML can be used to specify the system dataelements to be collected, the calculations to use to process the data,and when to produce performance measurement results.

At a high level, HPML is broken up into many different sub schemas thatrepresent the different parts of the overall HPML schema. Each part ofthe schema has different dependencies that work together to calculatemeasures and assessments on a given data source. The schema is separatedinto six distinct groups, 1) HPML, 2) Computation, 3) Results, 4)Assessments, 5) Measures, and 6) Instances and Periods. These groupsmake up the core components of HPML and can be added to or expanded withadditional links in the schemas. Each group is described in more detailbelow:

-   -   HPML: The HPML group refers to shared and top level elements        (e.g., MeasureDefinition element) and attributes that all the        other schemas must reference. This schema organizes all sub        schemas that make up the standard HPML Core.    -   Computation: The Computation group refers to the schemas that        represent the algorithms, triggers, and other computational        components of HPML. These components can be combined with both        the    -   Measures: The Measures group refers to schemas representing the        linking of data sources and computation to produce measurement        outputs from a given data source.    -   Assessments: The Assessments group refers to schemas        representing the assessment of a given measurement's values        either by category (Expert, Novice, etc.) or Value (100%, 99%,        75%, 10.3, etc.).    -   Results: The Results group refers to schemas representing the        output of both measures and assessments, detailing the        information produced by specific measure points throughout an        entire measurement period.    -   Instances and Periods: The Instances and Periods group refers to        schemas representing the creation and use of measures and        assessments for a given context. This schema defines the        instantiation of HPML elements at specific points in time, or        specific locations within space. Every element in this schema        has a time and/or location component. Whether that time is very        short or spans several years, whether the location is a small        area or a line on a map, the schema refers to when and where        data should be computed so that measures and results can be        linked to specific places and times. While the segregation of        HPML into multiple separate, but dependent, schemas may produce        an added level of complexity, ultimately, it provides the        language with the added benefit of extensibility, which is        critical in the ever changing technological landscape of        training environments. The modularity of HPML's design allows        for the replacement of sub-schemas with new additions while        still maintaining the core capability. Under this design, a        developer could extend the schema to allow for more        functionality, or to test new components for a specific context,        while keeping the base schema intact.

A full description of the HPML schema can be found in the HPML UserGuide posted to the HPML SISO Study Group. The User Guide includesdescriptions of elements of the schemas (e.g., definitions of what theelements and attributes mean) as well as example measures utilizingthese schemas.

For example and not for limitation, assume you are trying to measureaircraft performance in staying below a coordinating altitude (i.e.,altitude the aircraft cannot enter). Table 2 of FIG. 10 shows an exampleof a simple performance measure implemented in HPML. The DataRequestidentifies the object and its specific attributes that are required asinput to a measurement. In this case, the DataRequest is requesting theX, Y and Z components of the WorldLocation attribute of the Aircraft,which has an Id equal to a value that will be supplied later as aparameter. The MeasurementTemplate in our example uses the WorldLocationattribute of the Aircraft object to determine if the aircraft iscurrently above a prescribed altitude. The maximum altitude in thismeasure is defined as a constantValue of 4000 ft. in theMeasurementTemplate. A Parameter is also passed into theMeasurementTemplate, informing the measure for which specific Aircraftto retrieve the Altitude. The use of a Parameter here allows theMeasurementTemplate to be reused for different aircraft. At the core ofthe MeasurementTemplate are two nested MeasurementComputations. Theinner-most computation takes the DataRequest (WorldLocation.X,Y,Z) as anoperator and uses a customized function library to compute the Altitude.The result of this computation is then used as an operand for the outerMeasurementComputation, which compares the altitude value to aConstantRef that points to the maximum altitude ConstantValue anddetermines if the aircraft's altitude is greater than the prescribedmaximum altitude. The MeasurementDefinition finalizes the specificationcontained within the MeasurementTemplate. More specifically, theMeasurementDefinition defines any parameters that have been exposed, inthis case the marking identifier of the aircraft.

The example shown in Table 2 of FIG. 10 illustrates the basics of HMPL.Performance measures may be vastly more sophisticated and to includemany other elements such as assessment layers, training objectives andrelevant contextual information. Once expressed in HPML the performancemeasures will be available for use within the PM Engine. At simulationruntime, the HPML will be interpreted by the PM Engine. Based on theinformation defined in HPML, the PM Engine will connect to the requireddata sources, subscribe to the necessary data, use that data tocalculate performance measures and performance assessments and finallyoutput the resulting performance measurement and assessment data.

For embodiments for UAS students, feedback to the student may be basedon a set of carefully constructed performance measures. While thediscipline of the tactical environment will continue to limit thestudent's vocabulary, the ability to use speech and natural languagewill replicate real world mission environments and greatly enhance thestudent's learning through interaction with the tactical trainingsystem. The performance measures may be defined so that they provide thestudent with the information required for them to learn. This means thatthe definitions will be extended to include supporting measures,relevant contextual information, assessment criteria and otherinformation necessary for the student to self-learn.

Utterance Scoring.

In some embodiment, the communication training system is configured toassess or score utterances of the student. Embodiments of this systemutilizes as scoring mechanism that relies on the notion of slotalignment to align subsequences of words in the utterances of thestudent with exemplars of legal utterances based on each set ofsimulated test scenarios. The employed algorithm is a subset of theapproach developed by Sultan et al. A classification of each of thestudent's utterances is then used to score the utterances and generateappropriate feedback for training purposes.

The communication training system allows students to practice formationof proper communications for a variety of structured report types astraining events including SPOT, Battle Damage Assessment, RemoteHellfire, Call for Fire, Target Handover, and Close Air Support reports.

Each report or utterance type is composed of structured utterances whichcan be decomposed into phrases, or slots. In some cases, the phrases canbe further decomposed into smaller semantic units, e.g. a number orcount followed by a description of targets. Since we know the details ofeach simulated scenario a priori, we know what the student ought to bereporting. That is, we are afforded a very restricted domain ofappropriate utterances from the communication training system. Thismakes it possible to enumerate a set of expected utterances or exemplarphrases and semantic units for which the student should strive.Furthermore, we can classify the expected utterances into subsets wheresome phrases are more desirable than others. Using this partitioning ofphrases, we can apply a scoring mechanism that compares the phrases ofthe student's utterance to the subsets of expected utterances, allowingfor some phrases of the utterance to be preferred over others andgenerating a proportional score. The sets of expected utterances can beclassified in such a way that each subset of phrases has a similardeficiency, if a deficiency exists. This classification scheme can beexploited to generate constructive feedback when deficiencies in thestudent's utterance are encountered, e.g. the student should not referto the target as “a group of guys”, but instead use more precise andinformative language like “five individuals”.

Each simulated campaign mission is composed of training events withsimulated data presented to the student. The student is supposed toobserve the training event simulation data and respond with theappropriate utterance corresponding to the correct type of utterancereport(s) for the event. The information in the utterances has astructured form. For example, a well formed utterance for the SPOTreport includes descriptive observations along with the position andtime of the observations. FIG. 11A shows the information present in awell formed utterance for each of the report types. We refer to each ofthese elements of the utterance as a slot.

Each utterance's slot in FIG. 11A contains specific kinds of informationthat should have a reasonably structured form. Some slots have moredegrees of freedom in their expression than others. For example, thetarget location has a precise, standardized format which relaysinformation about the coordinates of the target; whereas targetdescription with counts can be formulated and uttered in many equallycorrect ways, and therefore requires a more sophisticated naturallanguage processing technique for assessment.

Subject matter experts (SME) may be used to predefine the common,anticipated slot phrases that a student may utter under thecircumstances presented in each of the simulated events. The SME maygroup the phrases into classes where some classes of phrases are moredesirable than others, i.e. the student should receive a higher scorefor uttering phrases in some classes than in other suboptimal classes.One of the classes should be reserved for optimal phrases, i.e. thereshould be a class of phrases that the student should be striving for andreceives the maximum score for uttering a phrase from this class. Eachof the other classes may be grouped such that the phrases in that classhave a common deficiency that make those phrases suboptimal, e.g. thestudent used less formal or descriptive language to describe the targetsuch as “a group of guys” instead of “five individuals”. Additionally,the classification system can be applied at any level of resolution thatis needed to sufficiently express the ranking of phrases. In otherwords, this system affords the definition of as many classes as arenecessary to properly rank the phrases under the simulated event towhich is pertains.

Note that the target description with count slot is common to all of thereport types. As the name suggests, this can be decomposed intosub-phrases. For example, the utterance, “five individuals carryingRPGs”, can be decomposed into a count of “five”, a target description of“individuals”, and a description of the activity they are engaged in,“carrying RPGs”. Decomposing the target description with count slot inthis way allows the enumeration of phrases by the SME to be simplifiedby enumerating the legal count/number phrases, target descriptionphrases, and activities phrases separately, reducing the slot to phrasesthat can be more easily enumerated independently. FIG. 11B shows afabricated example of this decomposition for illustrative purposes.

The SME's lists of common phrases with their classifications are encodedin a spreadsheet. The communication training system simulation softwareis designed to directly read the spreadsheet. This allows the SME andtrainers to work with a human readable format that the simulationsoftware can interpret to employ the rest of the scoring algorithm basedon the encoded information. Most importantly, this allows alterations oradditions to easily be made to the legal phrases and classificationsystem at any time, including after system delivery via text fileupdates.

Each training event in the communication training system hascorresponding utterance types that should be uttered by the student.Each training event's utterance type has an associated list of slotphrases as defined by the SME. The goal of slot alignment is to find theSME's slot phrase that most closely fits the student's actual utterance.This is accomplished through text alignment.

Text Alignment for Scoring Utterances.

Generally speaking, text alignment is the task of identifying textualsegments in different sources of text that have similar semanticmeaning. It is an important topic to many fields within natural languageprocessing including automatic machine translation, informationretrieval, question answering, and many others.

Text alignment is applied to a pair of phrases or utterances from a slotwhere one of the phrases is the student's utterance for that slot andthe other is an entry in the SME's list of expected utterance. There isa preliminary step performed before the main alignment algorithm isapplied. The location of stop words is identified in both phrases.Stop-words are common words in a language that often hold littlesemantic value and are often present to inform the reader/listener ofgrammatical structure, e.g. the, a, of, etc. Other words, not in thestop-word list, are considered to be content words. These content wordsare assumed to hold the more semantic meaning, and are therefore moreproductive for the purpose of aligning phrases.

The core of the text alignment algorithm is designed to find the longestcontinual subsequences, i.e. n-grams, of the content words in theshorter phrase that map to or otherwise align with subsequences in thelonger phrase. Since longer continual subsequences are preferred, thealgorithm searches for matching n-grams where n is initially equal tothe length of the shorter phrase, then n is reduced until either thereare no more matches, n equals zero, or all content words have beenaccounted for. In this way, a set of content alignments is producedbetween the two phrases.

An alignment scoring algorithm is used to score each pair of alignedphrases. The scoring algorithm, which is the Sorensen-Dice SimilarityMetric, is shown below:

${Score}_{alignment} = \frac{2{❘C❘}}{{❘A❘} + {❘B❘}}$

Where |A| is the number of content words in one phrase, |B| is thenumber of content words in the other phrase, and |C| is the number ofwords that were aligned. Note that |C|=|A∩B| when there does not existduplicate words in either phrase.

A text alignment score of the student's phrase is computed for each ofthe phrases in the predefined expected phrase list for the applicableslot. The predefined phrase with the highest score is selected as thebest aligned phrase for the slot and is used in for utterance scoringand feedback generation.

Determining Utterance Score.

There is a weight associated with each of the predefined expectedutterances, or utterance slot values, as shown in FIG. 11C. The weightnumerically encodes the correctness or acceptability of the phrasesbeing uttered in a given slot and is used in the scoring algorithm. Inthe embodiment shown, a higher weight means that the phrase is morecorrect than lower weights. Often, these weights are direct functions ofthe assigned class for the phrase, but the code is implemented in such away that this does not necessarily need to be the case. That is, if itbecomes convenient or necessary to customize the weights in the future,the code can be written to facilitate the changes to the spreadsheet.

An additional piece of information may be encoded in the slot'sdescription column. Since each event within a campaign mission reflectsa specific simulated training event which portrays a specific number oftargets that the student is reporting on, we encode the correct numberof targets for each description type. That is, if the description waspersons and the simulated event has five people depicted, we encode thatthere are five people. We call the correct number of targets theconstraint. This offers a way for the scoring and feedback modules tocompare the student's numeric description to the actual number ofspecific types of targets in the simulated event. If the student saysthere are more persons, for example, than exist in the simulation, thescore and feedback ought to reflect this mistake. Also, the number oftargets can be independently defined for individual types of targets.So, for example, we can encode an event where there are five people andtwo tanks by constraining the people descriptions to five and the tankdescriptions to two.

FIG. 11C shows examples of the weights and constraints. Note that FIG.11C is the same as FIG. 11B but with the additional weight andconstraint columns.

Scores are separately produced for two types of phrases. If the studentuttered “five persons carrying RPGs”, then the first scores phrases thatare aligned to a number or count and aligned to a target, e.g. “fivepersons” where “five” was aligned to the number and “persons” wasaligned to the target description. The second type of phrase is thealigned activity that the targets are performing, e.g. “carrying RPGs”.

The score for the count and target is a function of the weightsassociated with the number and target description, the presence of thenumber and target, and the numeric constraint. The utterance of acorrect cardinal number is directly rewarded, as specifying the definitenumber of targets in the simulator should be a goal of the student. Thescore for the count and target is shown below:Score_(target)=δ_(card)+δ_(cont) +k _(descr) ·w _(descr))/k _(norm)

Where δ_(card) and δ_(cont) are indicator functions for the presence ofa cardinal number and the case where the numeric constraint is notviolated, respectively. w_(descr) is the weight from FIG. 11C for thedescription, and k_(descr) is a coefficient that determines theimportance of the weight. k_(norm) is a normalizing constant so that thescore is linearly scaled between zero and one. In our case, k_(descr)=2,and k_(norm)=1+1+k_(descr)=4. The values of the coefficients can bechanged as desired to assign more weighting to the description element.

The score for the activity portion is simply the associated weightassigned in FIG. 11C. That is:Score_(activity) =w _(activity)

If multiple target or activity phrases are present in a student'sutterance, the scores above for each are averaged to produce a combinedscore. That is:

${Score}_{multiple} = {\frac{1}{N} \cdot {\sum\limits_{i = 0}^{N}{Score}_{i}}}$

These target and activity scoring methods are applied to all theappropriate report slots present in FIG. 11A.

Specific Event Measures and Performance Measures.

The set of performance measures may include the measurement andassessment of their communications as well as how their communicationsare coupled with their actions. This may include performance measuresthat assess the student's communications in the following ways:accuracy, completeness, timeliness, brevity, and order of individualcommunications; and appropriate coupling of communications to studentactions in the simulation. To support these different types of measures,as illustrated in FIG. 7C, the performance measures 782 may compriseaccuracy-content score data 782A, accuracy-form score data 782B,completeness score data 782C, timeliness score data 782D, order scoredata 782E, performance score data 782F and performance score algorithms782G. Measurement of these dimensions takes into account the content andform of student utterances, as well as contextual information from thesimulation environment. For example, accuracy of a description in a SPOTreport is relative to a known entity or event in the scenario capturedfrom the simulation environment, and timeliness is measured with respectto the event onset in simulation runtime.

Accuracy of communication encompasses both the content and form of apiece of student communication. That is, the system may decide: (1)whether a student's utterance expresses the content that is required ata given point in the scenario, and (2) whether the utterance meets therelevant protocols for military communication in this environment. Toevaluate content accuracy, the system compares expected utterancesdefined in the accuracy score data to observed student responses interms of semantic overlap. Did the student accurately describe andreport the event in the scenario? Student utterances must match one of aset of predefined possible lexical formulations for the event. Moreover,specificity counts: for example, “red truck” is likely preferred tosimply “truck”. Distinctions such as these are reflected in the accuracyscore. The expected content will be a representation of an optimalutterance at that point in the scenario. To compute semantic overlap,the system can make use of one or more semantic resources fordetermining semantic fit. For example, as described herein, one approachis to compute lexical overlap between the expected and observedutterances for each “slot” in the expected utterance template for theutterance type. This will permit variation in student utterances such assynonyms and paraphrases of the expected utterances (associating a costwith such variation, if desired). The variation in expression will bepredefined in the performance measures.

Evaluation of form is based on the evaluation of the content. Inparticular, evaluating form benefits from the correct identification ofa part of an utterance as expressing a particular kind of content andapplying form-based criteria to that part of the utterance. As withcontent analysis, the requirements for form may be derived fromavailable communications protocol, working with consultants, and/oranalyzing available data of training interactions. For example, giventhe variation of expression for form and content, in one embodiment anaccuracy event measure may be determined by aligning the utterance ofthe student with one or more of the utterance slots for the event typewhereby an utterance slot score can be determined. The utterance slotscore may be determined by a predefined score or other classificationassociated with the predefined utterance term or phrase that aligns withthe utterance of the student in that slot. An accuracy event measure forthe event type can be determined from the one or more utterance slotscore such as by summing all of the utterance slot scores for that eventtype.

Completeness measures the degree to which the student expresses all ofthe required items of information. This is measured principally bywhether the utterance fills all appropriate slots in the expectedutterance template for the event type. Did the student report allrequired information for the event? The completeness score data includesthe utterance template that predefines the type, slots, phrases, scoresand other data associated with the completeness event measure. Withinthe template, utterances are parsed into slots of required informationwith respect to communication type. For example, for a SPOT report as anevent type, slots include (1) number, (2) description, (3) activity, (4)location, (5) time, and (6) “what I'm doing”. Completeness may becomputed as the percentage of slots filled by the student. For example,in one embodiment, the completeness event measure is determined by themethod of aligning the utterance of the student with the one or moreutterance slot, determining whether the utterance slot is filled or notfilled by the utterance of the student, and determining the completenessevent measure as a percentage of the one or more utterance slot of theevent type filled by the utterance of the student.

Timeliness is assessed by considering how the student made proper use ofthe time that they had to communicate the message, and if it wascommunicated at the right time within the context of the mission. Thetimeliness score data includes the utterance template that predefinesthe type, slots, phrases, scores and other data associated with thetimeliness event measure. Note that timeliness does not always equate tofast, since it is important that the student understand the urgencyrelated to each communication and makes good use of the time afforded tothem so that they can form accurate and complete communications. Did thestudent report the event in a timely manner according to protocol?Timeliness is defined as the speed that a communication is formulatedand transmitted relative to event observation in the scenario. Forexample, in one embodiment, a timeliness event measure may be determinedby defining an utterance response time as the time between thepresentation of the simulation data to the student and the receipt ofthe response data and comparing that to expected utterance responsetimes. Each of the event types may have one or more expected utteranceresponse times aligned to a response time score and when the student'sutterance response time is aligned with an expected utterance responsetime the corresponding response time score defines the timeliness eventmeasure.

Evaluating order is based on protocol for the sequencing of particulardialog acts. This measure draws on the spoken interaction history of theinteraction manager and the NLP module's classification of utterancesinto expected dialog acts. The communications analysis module computeshow closely the ordering of information fits the optimal, prescribedordering (weighting the evaluation of performance by distance to theprotocol-based ordering, if desired). Did the order in which a studentreported the event match protocol? Most communication types must followa structured format where the order of slots of information isprescribed. The order score data includes the utterance template thatpredefines the type, slots, phrases, scores, order and other dataassociated with the order event measure. For example, in one embodiment,an order event measure is determined by aligning the utterance of thestudent with the one or more utterance slot and comparing the responsedata order to the expected utterance slot order to determine the orderevent measure. The expected utterance slot order may be predefined forthe utterance type and the utterance of the student defines a responsedata order reflecting the order of utterances of the student in theirresponse. The order may be computed as the distance in terms of “edits”(re-arrangement of a pair of slots) from the prescribed order.

A brevity event measure reflects whether the student reported the eventconcisely? Brevity can be operationalized in several ways. First,brevity may refer to the student's use of “brevity codes” at theappropriate times. The brevity score data includes the utterancetemplate that predefines the type, slots, phrases, scores and other dataassociated with the brevity event measure. For example, in oneembodiment, the utterance slot defines one or more brevity terms and theutterance of the student is aligned with the utterance slot and thebrevity term with it corresponding brevity score to determine one ormore utterance slot brevity score as the brevity event measure. If thereare more than one slots for that event type, the brevity scores for eachutterance slot can be summed to determine the brevity event measure.

A brevity event measure may also be operationalized as the speed or rateof transmission of the student's communication. For example, in oneembodiment, a time may be measured from the start of the utterance ofthe student and the completion of the utterance as the utteranceduration of the student. Each event type may have one or more predefinedexpected utterance durations with each of these corresponding to abrevity score. The utterance duration of the student may be aligned withthe expected utterance duration and the corresponding brevity score asthe brevity event measure.

Aa brevity event measure may also be operationalized as the “density” ofinformation conveyed-capture an intuitive notion of conciseness. Forexample, in one embodiment, a total number of words in the utterance ofthe student may be counted. Each event type may have one or morepredefined expected word counts with each of these counts correspondingto a brevity score. The word count of the student may be aligned withthe expected word count and the corresponding brevity score as thebrevity event measure.

Mission Performance Scoring.

In some embodiments, a scoring method is utilized to align and scoreutterances from student reports to determine a performance score and toprovide feedback to the student. The scoring framework generally “bins”performance measure scores such as event scores and then aggregates theminto scores for aligning with training events, missions, andsimulation-related points earned (toward “rank” promotion) as well asfor aligning with the execution data (e.g., state machine inputs).

For each performance measure type (i.e., result), measure triggers(i.e., when a measure should be calculated), measure components (e.g.,objects, attributes), and calculations to be performed on components toproduce the performance measure. In some embodiments, measures may bebinary (pass/fail) or stoplight (excellent/acceptable/deficient).

In one example embodiment, a performance scoring algorithm is definedfor each performance measure type as follows. For example, everyperformance measure type may get a green/yellow/red assessment (bin),for which performance thresholds (e.g., high/excellence=>X,med/acceptable=X, low/deficient/needing improvement=<X) are identified.The thresholds for each performance measure type are defined in aperformance measure type template, tailored for missions, as needed.Aggregating green/yellow/red measure scores at each level requiressetting a “passing” threshold (e.g., what % of possible score ispassing?), and determining whether weights (priority) should be appliedto any scores.

The use of stoplight scoring at each level requires a performancemeasure type to be defined as either pass/fail (Green/Red) ORGreen/Yellow/Red. For stoplight scoring, each training event must bepass/fail (Green/Red) and every Mission is pass/fail only. Rank isadvanced minimally for non-failing mission play. Increasing amounts ofexpert performance over time are required for higher ranks.

Determining the performance scoring algorithm for TEs involves anaggregation of behavioral and communications measures. Performancemeasure type bin scores are added together. The highest possible scoreis calculated, given the number of performance measure types (and anyweights) selected for that TE. If a particular performance measure typeis clearly higher-stakes than the others for that mission, weighting maybe applied. A pass/fail (green/red) threshold is defined for eachtraining event template (i.e., what % of possible score is a “passing”%), and a pass/fail score is calculated for each training event attemptbased on that threshold.

The performance scoring algorithm for Missions (aggregation of trainingevents) involves adding together the various training event scores forthat mission. Weighting is only used if the training event is new to thestudent—that's when the training event matters most (best opportunityfor developmental feedback). A pass/fail threshold is defined for eachMission (or across missions) using one of the following options:

-   -   Option 1: Certain number of training event attempts passed for        this mission (e.g., requires 4 of 6 TEs to pass this mission)    -   Option 2: What % of training event points is required for        “passing” a mission? (i.e., similar percentage-style calculation        with threshold as used in training event scoring)

Additionally, the system may apply a strategy for earning Career/Rankpoints (+points for good performance,—points for bad performance). Thesepoints may be used to determine Rank promotion criteria.

Performance Feedback Module (After Action Report (AAR)).

Performance feedback capability of the communications training systemmay be provided to include diagnostic performance assessments, includingmeasures of communication. Feedback to the learner is an important partof an effective training experience. In some embodiments, thecommunications training system supports feedback during task executionand in a final outbrief to the student. The performance feedback modulemay display assessments for both the individual tasks as well as anoverall assessment of key training objectives throughout the training.The performance feedback module may provide a presentation to thestudent indicating areas of improvement that can be made in his or herperformance with respect to timeliness, completeness, and communicationsdiscipline. The performance feedback module may indicate to the studentways in which specific scout and reconnaissance skills could be improvedvia tactical communications with specific units and entities such as theTOC and selected aviation units. In order to present a completeperformance feedback, the training environment may capture voicetransmissions from the student, translate voice to text, time stampcommunications and correlate voice transmissions to action within thesimulation.

The training environment performance feedback infrastructure may alsosupport display and review of newly available data derived from the PMEngine and access application module. This data may include bothconversational transcripts as well as metadata regarding theconversation (e.g., any failure modes that came about as a result of ASRfailures, terminated conversations, etc.). In addition, the performancefeedback module may incorporate display and review of measures ofperformance calculated by the PM Engine and access application modulealongside the measures calculated by the training environment system.

Feedback Generation.

The measurement environment is able to generate feedback on thestudent's utterances that may be specific to each event, report type,and slot. In the previous sections, the student's utterances weredecomposed and aligned with phrases in the SME's predefined legal phrase(utterance) lists. Each phrase in the list had an associated class. Theclasses can be used to group phrases together such that each class ofphrases has common deficiencies. Messages can be defined such that whena phrase with a certain deficiency is encountered, as defined by itsclass, an appropriate feedback message is displayed to the studentinforming them of the deficiency. The contents of the feedback messagecan be defined in concert with the definition of the classes. Thisaffords the ability to alter or tailor the feedback messages in thefuture by forming new classes and constructing appropriate messages thatpertain to the simulated event that the student is practicing.

In addition to using the classes to provide feedback, other types offeedback may be generated. For example, in the number and target portionof a slot, the numeric constraint can be used to generate feedback ifthe student mentioned more targets than exist in the simulated event.Also, the absence of phrases can trigger feedback informing the studentof the missing information. In other words, if either δ_(card) orδ_(cont) are equal to zero during the utterance scoring, feedback can begenerated to inform the student that they should use cardinal numbers tospecify the number of targets or that they have specified more targetsthan exist in the simulated event.

The performance feedback module is implemented by producing a list offeatures while a slot is being analyzed in the utterance scoring sectionof the code. Features essentially indicate properties of a phrase thatmay elicit feedback, such as the student exceeded the numeric constraintor a portion of the phrase is missing. Combining the features with theclasses of aligned phrases provides a structured way to selectappropriate and constructive feedback to the student. FIG. 11D shows anexample of features, classes, and the corresponding feedback message,and is intended to work with the information defined in FIG. 11C.

Note that in FIG. 11D, an artificial class of zero was introduced toindicate the absence of that information from the utterance. Also,feedback messages of N/A indicate that there was no deficiency with thisfeature and therefore, no constructive feedback is necessary. As anexample, if the student uttered, “five persons carrying RPGs”, thephrase would be decomposed into a target count of “five”, a targetdescription of “persons”, and an activity of “carrying RPGs”. Thestudent has uttered all relevant portions, the number is the correctcardinal number according to the constraint, and each portion belongs toclass 1 which in this case corresponds to generating no feedback. But ifthe student instead uttered, “Guys looking suspicious”, they are missingthe number of targets and they used a class 3 target description andactivity. This would trigger a NoNumber feature, a class 3DescriptionClass feature, and a class 3 ActivityClass, which in turngenerates the feedback, “The number of target persons was missing”, “Thecorrect description was persons, people, or individuals. You shouldavoid using informal language. Specific descriptions of persons involvedin the activity provide better context from which to make decisions”,and “Your description of the activity of the enemy was too vague”,respectively

Other Performance Measures.

Measures may also include measures of usability, utility, andeffectiveness of the tactical communications. These measures may includereflecting the correct level of fidelity to replicate the operatingenvironment, accuracy and tone of the communications from syntheticentities, ease of use, learner engagement, technical or doctrinalaccuracy. Methods of obtaining useful measurements for these measuresmay include structured interviews with both novice and expert users andthe use of standardized surveys.

Other measures may also include pitch/tone of voice, amplitude of voice,hand gestures, or clarity of speech through pronunciation or words oroccurrences of hesitation words (“um”, “ah”, etc.).

One Embodiment of a Communication Training System

One embodiment of training communication systems generally comprises thefunctional elements of FIGS. 1, 3A and 7A implemented in a softwareprogram product to be executed by a computer or processor based system.The computer or processor based system may be a generic computer, aspecifically programmed computer, a specifically programmedcomputer-based training simulator or a computer-based training simulatorinteracting with other generic or specifically designed/programmeddevices.

As will be readily apparent to those skilled in the art, trainingcommunication systems and methods can be embodied in hardware, software,or a combination of hardware and software. For example, a computersystem or server system, or other computer implemented apparatuscombining hardware and software adapted for carrying out the methodsdescribed herein, may be suitable. One embodiment of a combination ofhardware and software could be a general purpose computer system with acomputer program that, when loaded and executed, carries out therespective methods described herein. In some embodiments, a specificallydesigned computer-based training system, containing specialized hardwarefor carrying out one or more of the instructions of the computerprogram, may be utilized. In some embodiments, the computer system maycomprise a device such as, but not limited to a digital phone, cellularphone, laptop computer, desktop computer, digital assistant, server orserver/client system.

Computer program, software program, program, software or program code inthe present context mean any expression, in any language, code ornotation, of a set of instructions readable by a processor or computersystem, intended to cause a system having an information processingcapability to perform a particular function or bring about a certainresult either directly or after either or both of the following: (a)conversion to another language, code or notation; and (b) reproductionin a different material form. A computer program can be written in anyform of programming language, including compiled or interpretedlanguages, and it can be deployed in any form, including as astand-alone program or as a module, component, subroutine, or other unitsuitable for use in a computing environment.

FIG. 6 is a schematic diagram of one embodiment of a computer system 600by which the disclosed methods for communications training may becarried out. The computer system 600 can be used for the operationsdescribed in association with any of the computer implemented methodsdescribed herein. The computer system 600 includes at least oneprocessor 610, a memory 620 and an input/output device 640. Each of thecomponents 610, 620, and 640 are operably coupled or interconnectedusing a system bus 650. The computer system 600 may further comprise astorage device 630 operably coupled or interconnected with the systembus 650.

The processor 610 is capable of receiving the instructions and/or dataand processing the instructions of a computer program for executionwithin the computer system 600. In some embodiments, the processor 610is a single-threaded processor. In some embodiments, the processor 610is a multi-threaded processor. The processor 610 is capable ofprocessing instructions of a computer stored in the memory 620 or on thestorage device 630 to communicate information to the input/output device640. Suitable processors for the execution of the computer programinstruction include, by way of example, both general and special purposemicroprocessors, and a sole processor or one of multiple processors ofany kind of computer.

The memory 620 stores information within the computer system 600. Memory620 may comprise a magnetic disk such as an internal hard disk orremovable disk; a magneto-optical disk; an optical disk; or asemiconductor memory device such as PROM, EPROM, EEPROM or a flashmemory device. In some embodiments, the memory 620 comprises atransitory or non-transitory computer readable medium. In someembodiments, the memory 620 is a volatile memory unit. In otherembodiments, the memory 620 is a non-volatile memory unit.

The processor 610 and the memory 620 can be supplemented by, orincorporated in, ASICs (application-specific integrated circuits).

The storage device 630 may be capable of providing mass storage for thesystem 600. In various embodiments, the storage device 630 may be, forexample only and not for limitation, a computer readable medium such asa floppy disk, a hard disk, an optical disk, a tape device, CD-ROM andDVD-ROM disks, a “thumb” drive, alone or with a device to read thecomputer readable medium, or any other means known to the skilledartisan for providing the computer program to the computer system forexecution thereby. In some embodiments, the storage device 630 comprisesa transitory or non-transitory computer readable medium.

In some embodiments, the memory 620 and/or the storage device 630 may belocated on a remote system such as a server system, coupled to theprocessor 610 via a network interface, such as an Ethernet interface.

The input/output device 640 provides input/output operations for thesystem 600 and may be in communication with a user interface 640A asshown. In one embodiment, the input/output device 640 includes akeyboard and/or pointing device. In some embodiments, the input/outputdevice 640 includes a display unit for displaying graphical userinterfaces or the input/output device 640 may comprise a touchscreen. Insome embodiments, the user interface 640A comprises devices such as, butnot limited to a keyboard, pointing device, display device or atouchscreen that provides a user with the ability to communicate withthe input/output device 640.

The computer system 600 can be implemented in a computer system thatincludes a back-end component, such as a data server, or that includes amiddleware component, such as an application server or an Internetserver, or that includes a front-end component, such as a clientcomputer having a graphical user interface or an Internet browser, orany combination of them. The components of the system can be connectedby any form or medium of digital data communication such as acommunication network. Examples of communication networks include, e.g.,a LAN, a WAN, wireless phone networks and the computers and networksforming the Internet.

One example embodiment of the systems and methods for communicationstraining may be embodied in a computer program product, the computerprogram product comprising a computer readable medium having a computerreadable program code tangibly embodied therewith, the computer programcode configured to implement the methods described herein, and which,when loaded in a computer system comprising a processor, is able tocarry out these methods.

One embodiment of a computer or processor based system forcommunications training is shown in FIG. 7A. As shown, the trainingenvironment 710 comprises the environment that the students may be usingfor their training. The environment may comprise one or more studentsinteracting with a computer-based simulator 740 as the simulationenvironment through multiple tasks of scenario missions. The studentsmay interface with the computer-based simulator utilizing multiplemethods. The student may interface with the simulator directly utilizinga user interface 720D of the simulator, through a user interface device720A communicating directly to the simulator or through a user interfacedevice (720B and 720C) over a network to all or portions of thecomputer-based simulator 740. The students may interface independentlyor they may interface simultaneously and they may interface in adistributed training environment over a distributed data network. Thecomputer-based simulator 740, or the simulator software is comprised ofthe following integrated modules and services that function within thesimulator or across a communication network to generate the missiontraining environment: simulation dataset 757, the interaction managermodule 758, the communication platform modules 752 and the userinterface 720D. The computer-based simulator 740 may allow the user tointerface with the user interface 720D, a synthetic entity module 750, aperformance dataset database 766A and/or a training content database742A. The synthetic entity may utilize communication platform modules752, a simulation dataset database 757 and an interaction manager module758 to provide communication with the user. The performance measurementplatform 780 may communicate with the training environment 710 and aperformance measure database 782 to provide performance measures to thecomputer-based simulator 740 and/or the student.

Within the computer-based simulator 740, a selected trainingcontent/scenario from the training content database 742A is used by theinteraction manager module 758 to dictate the scenarios presented to theuser though the user interface. The scenarios include communicationsfrom the simulation dataset 757 as well as any other requiredconfigurations for the user interface. The training content/scenarios742A are communicated to the user through the user interface (720A-720D)and may include communications defined by the simulation dataset 757from the communication platform modules 752. The communications from thestudent to the user interface is communicated to and received by thesynthetic entity 750 though the communication platform modules 752.Within the functions of the synthetic entity 750, the communicationplatform modules 752 transform the verbal communication to text andcommunicate the text to the interaction manager module 758. Theinteraction manager module 758 attempts to align the text received toentries in the training content 742A such as the execution dataset 759and performance measures 782. This alignment is used to identify thecommunication received and compare that to the communication expectedfor the training content/scenario selected. The alignment of the textreceived to the execution dataset is used to determine the state of thestudent and the simulation to determine what action should be nextperformed by the simulator. The alignment of the text received to theperformance measures 782 is used to determine an event measure of thestudent in the simulation. The communication received may also be storedin a performance dataset database 766A to be used to compare thecommunication received to performance measurement data. The performancedataset may also be communicated to the performance measurement engineserver 780 for comparison to a performance measures database 782 tomeasure the performance of that communication against predefinedperformance measures. Components of the above system may be co-locatedor they may be distributed over a communication network. For example, asshown, the performance measures 782, the training content database 742Band/or performance dataset database 766B may not be in thecomputer-based simulator 740 but may be in communication with thesimulator 740 over a data network.

One Example Embodiment of a Communications Training System in Operation

For illustration purposes and not for limitation, the operation of anexample embodiment of a communication training system consistent withFIG. 3A will be discussed. This embodiment, a NVTT simulator, progressesthe student through a series of UAS scout mission scenarios involvingvarious assets (e.g., manned, unmanned, aerial, and ground-based). Thestudent, while seated at a laptop GCS, interacts with virtual teamentities via ad hoc communication over a simulated radio to accomplishincreasingly complex missions. NVTT is a web-based system integratingnatural language processing components (i.e., speech recognition,speech-to-text, text-to-speech, and language recognition), performancemeasurement and feedback modules into a One Semi-Automated Forces(OneSAF) flight simulator platform.

To build the training content for this simulator, Aircrew TrainingManual (ATM) tasks were mapped to the 10 training missions and a reviewof missions led to alignment of campaign missions to student events bysystem type (Shadow or Grey Eagle). Activity diagrams were created todescribe the action between the student and the constructive entities aswell as the branches and sequels in the action for: Indirect fire, closecombat attack, target handover with and without the LTM and LDRF andremote Hellfire designation.

To build the execution data for the simulator, the activity diagramswere used to populate the state transition tables for the statemachines.

To build on existing simulation data, completed modifications were madeto to OneSAF campaign mission scenarios to account for the simulationsuse of audio and text data. Mission scripts were created for campaignmissions with recommended injects providing students with additionmission situational awareness. A crosswalk was done to ensure alignmentof utterances for scenario missions, events, andcommunications/utterance formats.

Referring to FIG. 8A, in operation, generally the communication trainingsystem receives a communication data at 830, transforms thecommunication data to a text, event data at 840 and at 851, the systemaligns the communications received to an expected response data todetermine an event measure. The event measure may be the performancemeasure or multiple events may be used to determine a performancemeasure. And given the event measure from 851, a performance measure isdetermined at 882. These features are enabled by the inclusion of, andthe special formatting of the datasets used to make these comparisonsand assessments. In particular, the predefinition of utterancetemplates, with expected utterances and corresponding scores for eachutterance type, allow for a broad range of responses to result in abroad range of measures to assess the student's performance.

In operation, the system generally allows the user or the system toselect a training scenario at 810. With this scenario selected andcommunicated to the training system, the synthetic entity is then ableto, through the user interface of the system, present the scenario tothe user at 820 and receive a communication back from the user as aresponse data at 830. This communication is transformed to text data at840 and communicated to the interaction manager module to makecomparisons to different data step at 850 and 851. The interactionmanager module receives the communication data and aligns it withexpected input data for the state machines 852 to see whether theresponse is one of the expected response. If the response aligns withone of the expected responses, at 854 the response is compared to stateand transition values in the state model to determine the state of thestudent and/or the scenario at 856 and determine whether the studentand/or the scenario should transition to another action at 858. With orwithout determining state at 850, the system takes the transformedcommunication from 840 and measures the event at 851 by aligning theutterance to a predefined utterance type at 853, based on the utterancetype the student's utterance is aligned to slots at 855. The utteranceslots include predefined phrases or utterance values with correspondingvariables such as scores so that when the student's utterance is matchesto the predefined phrase or utterance value, a corresponding utterancescore is determined for that utterance slot at 857. This utterance slotscore, along with any other utterance slot scores for that utterancetype, is used to determine the utterance type score at 850. The eventmeasure from 851 is used to determine a performance measure at 883 andthe performance measure is assessed against assessment algorithms at 884to determine a performance assessment. Feedback based on the eventmeasure, the performance measure and/or the performance assessment maybe provided at 886. In some embodiments, the event measure, performancemeasure, performance assessment or other feedback may be provided to alearning management system at 870 for additional analysis such as butnot limited to subsequent scenario selection by the training system.FIGS. 8B-8E illustrate further details of the operation of acommunications system trainer consistent with FIG. 8A.

Referring to FIG. 8B, in some embodiments the step of selecting atraining scenario at 810 may comprise defining the training event 812,defining the simulation data 814 necessary to execute the event anddefining the expected performance data 816 for this scenario. Thisinformation may be received from the training content database andcommunicated with the training system, in particular with the syntheticentity, for use during the training simulation.

Referring to FIG. 8C, in some embodiments the step of transforming thecommunication 840 may comprise receiving the communication at 842,transforming the communication to text data at 844 and formatting thedata to text data 846 in a format that can be used for aligning with thevalues in the state machines and the predefined utterance templates.

Referring to FIG. 8D, in some embodiments the step of aligning the textdata to a transition data at 854 may comprise receiving the statemachine data at 854A which may be received from the training contentdatabase. The text data representing the student's response can then bereceived at 854B from the interaction manager and the text data valuescan be aligned with the predefined transition data value to determinethe state of the student and other system components.

Referring to FIG. 8E, in some embodiment the step of measuring an eventat 851 comprises measuring an utterance type as an event type. In thisembodiment, the utterance is received as the event data and thisutterance is aligned to an utterance type at 853. The utterance typedefines the utterance slots according to the predefined utterancetemplates. At 855, the utterances within the event data are aligned tothe slots of that utterance type and at 857 a score is determined forthe utterance slots. The score for the utterance slot is determined byaligning the actual utterance with a predefined utterance and using thescore associated with that predefined utterance as the slot score. Anutterance type score is determined from the event data and the utterancescores at 859. The utterance type score may be one of the event measuressuch as accuracy, completeness, timeliness, order or brevity.

Referring to FIG. 8F, in some embodiments the step measuring performance882 may comprise receiving the performance assessment measures at 882Afrom the PM Engine, receiving the utterance type score at 882B anddetermining the performance score at 882C based on the assessmentmeasure and the utterance type score. For example, the utterance typescores may be numeric score as defined for that utterance type and theperformance assessment measure for that utterance type may be agreen/yellow/red score based on the numeric score for that utterancetype where a score of 80 or above is green, a score between 60 and 79 isa yellow and a score of 59 or below is a red. In this example, theperformance score may be determined by comparing the utterance typescore to the performance assessment score. In embodiments that assessthe performance of multiple events, performance assessment scores may bedetermined by more involved algorithms. One such example may be analgorithm requiring a percentage of utterance type scores needing toexceed a threshold number of yellow scores.

FIG. 4A illustrates a set of state machines that represent coordinatedsteps required to execute close combat attack and require both NVTT coreand Voisus to share state updates as the situation unfolds. Thecoordinated steps outline base cases as well as alternative andfailure/recovery modes for target handover and call for fire scenarios.Base cases represent sunny-day/error-free scenarios. The common basecase for target handovers applies to handovers from student (as apayload operator) to ground and rotating wing air platforms and ispictured in FIG. 4A. As noted, there are a large number of failure andrecovery modes that these scenarios have been grown to handle. Thosefailure modes are themselves represented as nested state machines.Nested state machines that allow recovery from failure modes get graftedinto the base case at the appropriate transition in FIG. 4A markedG_(x).

FIG. 5 also illustrates the processes involved in state synchronizationof the interaction manager module (MissionExecution) to keep thecommunication platform state (here Voisus) consistent with the state ofthe simulation data (here OneSAF) so that the event information sharedwith the student through the audio interface (here Voisus client) isconsistent with the event information shared with the student throughthe other simulator interfaces (here NVTT client).

Table 2 of FIG. 10 depicts the processes involved in enabling thisstepwise progression. Note that both ConstructProxy::Voisus andSAFProxy::UDG are HTTP connections and as such operate outside of thesimulation bus (DIS in the current case) consistent with FIG. 5 . As thestudent progresses through the verbal communications required as part ofthe cooperative engagement with the synthetic entity (in this caseApache, but the same holds true for Stryker or other ground entities),the MissionExecution (as the interaction manager) continuously collectsstate changes from ASTi through its ConstructProxy and reconciles themwith state it manages as it executes SAF behaviors and monitors statecoming out of the SAF via SAFProxy. The result is a step-wiseprogression through the coordinated state machines defined for targethandover, whether it be with or without weapons, or with one of theweapons supported by the coordinating platform. Depending on the NVTTmission type (training versus campaign) either NVTT's LMS or DDSMcomponents are responsible for this reconciling of state and stimulationof OneSAF.

The tables shown in FIGS. 9 and 10 are extracted from the NVTT statemachine documentation and follow the sunny-day target handover scenariooutlined in FIG. 4A.

Feedback Forms.

At the conclusion of each mission, the student may receive tailoredfeedback in several forms based on the performance measure results suchas those listed below.

Stoplight indicator feedback: Green, red (and yellow, when applicable)indicators of progress on specific missions, training objectives, andmeasures.

Narrative feedback: canned text generated from patterns of scores withina mission/training objective; feedback contains description ofperformance plus recommendations for doing better. The feedback contains(a) 1-2 sentences on what the expected competencies were for thisreport; (b) short list of bullets of how the student's report wasdeficient; (c) instructions to listen to the Sample Report to see howthat could have been reported better.

Audio report comparison feedback: listening to your report vs. an expert(Sample report); provides comparisons for cadence, emphasis, and claritythat narrative feedback shouldn't have to handle.

Although this invention has been described in the above forms with acertain degree of particularity, it is understood that the foregoing isconsidered as illustrative only of the principles of the invention.Further, since numerous modifications and changes will readily occur tothose skilled in the art, it is not desired to limit the invention tothe exact construction and operation shown and described, andaccordingly, all suitable modifications and equivalents may be resortedto, falling within the scope of the invention which is defined in theclaims and their equivalents.

We claim:
 1. A computer-based communications training simulatorconfigured to determine a performance measure for a student, thecomputer-based communications training simulator comprising: a memoryconfigured to store a training content data set comprising a trainingevent data; the training event data comprising a simulation data and anevent type; a communication platform configured to present thesimulation data to a student and receive a response data of the studentto the simulation data; each of the response data comprises an actualutterance of the student; the communication platform configured toreceive a first response data of the student and transform the firstresponse data to a first text data; an interaction manager moduleconfigured to receive the first text data to determine a first eventdata; the communication platform configured to receive a second responsedata of the student and transform the second response data to a secondtext data; the interaction manager module configured to receive thesecond text data to determine a second event data; the interactionmanager module configured to determine a first event measure for thestudent based on the first event data and the event type; theinteraction manager module configured to determine a second eventmeasure for the student based on the second event data and the eventtype; a measurement environment configured to execute a performancescore algorithm to determine a performance score from the first eventmeasure and the second event measure as a performance measure for thestudent; wherein the event type comprises an utterance type; wherein themeasurement environment comprises a predefined utterance scoring datacomprising: the utterance type defining an utterance slot, the utteranceslot corresponding to a plurality of utterance slot values and eachutterance slot value corresponding to an expected utterance of thestudent, and the utterance slot value also corresponding to an utteranceslot score; and wherein the interaction manager module is configured to:align the first event data of the student to a first utterance slotvalue to define a first utterance slot score as the first event measurefor the student, and align the second event data of the student to asecond utterance slot value to define a second utterance slot score asthe second event measure for the student.
 2. The computer-basedcommunications training simulator of claim 1 wherein the measurementenvironment further comprises a predefined performance scoring datacorresponding to the first event data and the second event data.
 3. Thecomputer-based communications training simulator of claim 1 wherein theperformance score from the first event measure and the second eventmeasure comprises a sum of the first event measure and the second eventmeasure.
 4. The computer-based communications training simulator ofclaim 1 wherein the interaction manager module is further configured topresent an audio data to the student based on the simulation data andthe response data.
 5. The computer-based communications trainingsimulator of claim 1 wherein the performance measure comprises anaccuracy event measure.
 6. The computer-based communications trainingsimulator of claim 1 wherein the performance measure comprises acompleteness event measure.
 7. The computer-based communicationstraining simulator of claim 1 wherein one of the first event measure orthe second event measure comprises a timeliness event measure.
 8. Thecomputer-based communications training simulator of claim 1 wherein oneof the first event measure or the second event measure comprises abrevity event measure.
 9. The computer-based communications trainingsimulator of claim 1 wherein the performance score from the first eventmeasure and the second event measure comprises an order event measurebased on a sequence of the first event measure and the second eventmeasure.
 10. The computer-based communications training simulator ofclaim 1 wherein: the interaction manager module is further configured toalign the first event data of the student to a first utterance slot todefine a first utterance slot score as the first event measure for thestudent; the interaction manager module is further configured to alignthe second event data of the student to a second utterance slot todefine a second utterance slot score as the second event measure for thestudent; and the interaction manager module is configured to align thefirst event data and second event data of the student to the first andthe second utterance slot according to an utterance slot score metricdefined as: ${Score}_{alignment} = \frac{2{❘C❘}}{{❘A❘} + {❘B❘}}$wherein: |A| is a number of content words in an utterance slot, |B| is anumber of content words in an event data, |C| is a number of words thatare aligned, and the event data with a highest utterance slot score isselected as a best aligned event data for the utterance slot.
 11. Acomputer-based communications training simulator configured to determinea performance measure for a student, the computer-based communicationstraining simulator comprising: a memory configured to store a trainingcontent data set comprising a training event data; the training eventdata defining a plurality of simulation data and an event type; a userinterface configured to present the simulation data to a student andreceive a response data of the student to the simulation data; acommunication platform configured to receive the response data of thestudent and transform the response data to a text data; an interactionmanager module configured to receive the text data to determine an eventdata; the event type comprises an utterance type and the response datacomprises an actual utterance of the student; a measurement environmentcomprises a predefined utterance scoring data comprising: the utterancetype defining an utterance slot, the utterance slot corresponding to aplurality of utterance slot values and each utterance slot valuecorresponding to an expected utterance of the student, and the utteranceslot value also corresponding to an utterance slot score; theinteraction manager module is further configured to align the event dataof the student to the utterance slot value to define the utterance slotscore as an event measure for the student; and the measurementenvironment further comprises: a predefined performance scoring data, aperformance data comprising the event measure for the student, and aperformance score algorithm configured to determine a performancemeasure for the student based on the performance data and the predefinedperformance scoring data.
 12. The computer-based communications trainingsimulator of claim 11 wherein the interaction manager module is furtherconfigured to present an audio data to the student based on thesimulation data and the response data.
 13. The computer-basedcommunications training simulator of claim 11 wherein the performancemeasure of the student comprises one event measure selected from thegroup consisting of: an accuracy performance measure; a completenessperformance measure; a timeliness performance measure; a brevityperformance measure; and an order performance measure.
 14. Acomputer-based communications training simulator configured toautomatically determine a next simulation to present a student, thecomputer-based communications training simulator comprising: a memoryconfigured to store a training content data set comprising a simulationdata and an event type; the event type defining a transition data basedon an event data and a current state of the student; a user interfaceconfigured to present the simulation data to a student having thecurrent state of the student; the user interface further configured toreceive an actual utterance of the student to the simulation data; acommunication platform configured to receive the actual utterance of thestudent and transform the actual utterance to a text data; and aninteraction manager module configured to: receive the text data as theevent data, and define a next simulation data to present to the studentbased on the event data, the current state of the student and thetransition data; wherein the interaction manager module is furtherconfigured to align the event data of the student to an utterance slotto define an utterance slot score as an event measure for the student;and the interaction manager module is configured to align the event dataof the student to the utterance slot according to an utterance slotscore metric defined as:${Score}_{alignment} = \frac{2{❘C❘}}{{❘A❘} + {❘B❘}}$ wherein: |A| is anumber of content words in the utterance slot, |B| is a number ofcontent words in the event data, |C| is a number of words that arealigned, and the event data with a highest utterance slot score isselected as a best aligned event data of the student to the utteranceslot.
 15. The computer-based communications training simulator of claim14 wherein the next simulation data comprises an audio data.
 16. Thecomputer-based communications training simulator of claim 14 wherein thetransition data comprises a transition table relating the event data,the current state of the student and the next simulation data.
 17. Thecomputer-based communications training simulator of claim 14 wherein theinteraction manager module is further configured to align the event dataof the student to an utterance slot value to define an utterance slotscore as an event measure for the student; and the interaction managermodule is further configured to determine the current state of thestudent based on the event data.
 18. The computer-based communicationstraining simulator of claim 14 wherein the event measure of the studentcomprises one event measure selected from the group consisting of: anaccuracy performance measure; a completeness performance measure; atimeliness performance measure; a brevity performance measure; and anorder performance measure.